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Rating the relevance of QUORUM-selected ASRS incident narratives 
to a "controlled flight into terrain" accident 

MICHAEL W. MCGREEVY AND IRVING C. STATLER 
NASA Ames Research Center 


Summary 

An exploratory study was conducted to identify 
commercial aviation incidents that are relevant to a 
"controlled flight into terrain" (CFIT) accident using a 
NASA-developed text processing method. The 
QUORUM method was used to rate 67820 incident 
narratives, virtually all of the narratives in the 
Aviation Safety Reporting System (ASRS) database, 
according to their relevance to two official reports on 
the crash of American Airlines Flight 965 near Cali, 
Colombia in December 1995. For comparison with 
QUORUM’S ratings, three experienced ASRS analysts 
read the reports of the crash and independently rated 
the relevance of the 100 narratives that were most 
highly rated by QUORUM, as well as 100 narratives 
randomly selected from the database. 

QUORUM successfully retrieved a large proportion of 
incidents that are relevant to the Cali accident. Eighty - 
four of 100 QUORUM-selected narratives were rated 
as relevant to the Cali accident by one or more of the 
analysts. Each analyst rated approximately two-thirds 
of the QUORUM-selected narratives as relevant. Over 
two-thirds of the incidents retrieved by QUORUM 
were rated as relevant to the context, events, 
problems, or human factors of the Cali accident. Over 
half of the incidents were rated as relevant to the 
causes of the accident. In addition, the QUORUM 
collection of incidents was found to be significantly 
more relevant than the random collection. 

The 84 incidents that were rated as relevant to the Cali 
accident involved a variety of factors, including over- 
reliance on automation, confusion and changes during 
descent/approach, terrain avoidance, and operations in 
foreign airspace. 

These results show that a QUORUM-derived query 
model based on accident reports can be used 
successfully to retrieve relevant incident reports. 

Introduction 

The relevance of commercial aviation incidents to 
accidents is not firmly established by hard data, but 
most analysts believe that incidents and accidents are 
importantly related. Aviation accidents, which are 


relatively rare, involve major injury to persons, 
significant damage to property, or both. Incidents are 
more common, but do not result in significant damage 
or injury. Many aviation experts (e.g., Miller, 1998a; 
Miller, 1998b) have long felt that accidents generally 
involve chains of causation among otherwise 
relatively minor factors that, taken together, lead to 
serious consequences. These minor factors, occurring 
in isolation, might otherwise result in an incident. By 
identifying accident factors that also occur in 
numerous incidents, it may be possible to break 
critical links in the chain of causation. 

The Cali accident 

On the night of December 20, 1995, American 
Airlines Flight 965, a Boeing 757 carrying 163 
passengers and crew on a regularly scheduled flight 
from Miami, Florida to Cali, Colombia, crashed into 
the side of a mountain, just short of its summit 
(NTSB, 1996a; NTSB 1996b; McKenna, 1996a; 
McKenna, 1996b; Kaiser, 1996). Only 4 passengers 
survived the crash. This crash is classified as a CFIT 
(controlled flight into terrain) accident. According to 
Duke (1996), "CFIT remains the leading killer of 
airliner occupants, far exceeding windshear or midair 
collisions." For that reason, preventing CFIT accidents 
is a high priority (Shifrin, 1998; McKenna, 1998a; 
McKenna, 1998c; Anon., 1998; Roberts, 1996; Scott, 
1996). 

Because the accident happened in Colombia, it was 
investigated by Aeronautica Civil of the Government of 
Colombia. In accordance with international protocol, 
assistance was provided by the U.S. National 
Transportation Safety Board (NTSB), the U.S. Federal 
Aviation Administration (FAA), American Airlines, the 
Allied Pilots Association, Boeing Commercial Airplane 
Group, and Rolls Royce, Inc. In September, 1996, 
Aeronautica Civil issued its report (Aeronautica Civil, 
1996). In response to the accident and the Colombian 
report, the U.S. National Transportation Safety Board 
issued recommendations in the form of a letter to the 
Federal Aviation Administration, dated October 16, 1996 
(NTSB, 1996c). 



These two official government accident report 
documents cite a number of factors that contributed to 
the Cali accident. These factors include, but are not 
limited to: 

• execution of a GPWS (ground proximity warning 

system) escape maneuver without retracting speed 
brakes and without angle of attack information 
other than the stick shaker, 

• loss of situational awareness by the flight crew, 

• failure to recognize loss of situational awareness, 

• additional workload imposed by a runway change 

during an automation-aided approach, 

• the effect of time pressure on execution of 

procedures, 

• over-reliance on the FMS (flight management 

system), 

• the loss of waypoint data when a direct clearance 

was entered into the FMS, 

• the potential for confusion when non-standard 

names are used for standard terminal arrival 
routes, 

• the potential for confusion of two proximal 

navigation beacons having the same identifier, 

• use of the wrong navigational beacon, 

• an unintended aircraft maneuver, 

• inadvertent course deviation, 

• miscommunication between flight crew and 

controller (especially given the language and 
cultural differences), 

• over-reliance on ATC (air traffic control), 

• problems with crew decision-making, 

• the lack of commonality between instrument 

approach charts and FMS displays, 

• the lack of detailed terrain information on approach 

charts, 

• the lack of terrain information on FMS displays, 

• the lack of radar coverage when accustomed to 

having it, 

• possible problems associated with commercial 

aviation operations in Latin America. 

ASRS incidents 

While commercial aviation accidents are relatively 
infrequent and typically involve seemingly unique 
combinations of circumstances, commercial aviation 
incidents are more common and can sometimes be seen to 
fail into patterns. Since the causes of incidents are thought 


to be related to the causes of accidents, considerable 
effort has gone into collecting incident reports for 
analysis of Tieir safety implications. Many aviation safety 
databases are currently in operation, each with its own 
purposes and characteristics. (For examples of U.S. 
government databases, see FAA, 1998). The Aviation 
Safety Reporting System (ASRS) database is of particular 
interest because it contains a large number of narratives 
written by those who directly observed, and usually 
participated in, the incidents themselves. 

According to the Federal Aviation Administration (FAA, 
1998), "The Aviation Safety Reporting System (ASRS) is 
a voluntary confidential and anonymous incident 
reporting s> stem. It is a cooperative program established 
under FAA Advisory Circular No. 00-46D, funded by the 
FAA and administered by NASA. Information collected 
by the ASRS is used to identify hazards and safety 
discrepancies in the National Aviation Airspace System. 

It is also used to formulate policy and to strengthen the 
foundation of aviation human factors safety research." 

The ASRS has been a major clearinghouse of commercial 
aviation incident reports for over 20 years (Reynard, 
Billings, Cheaney, and Hardy, 1986; ASRS, 1998). The 
ASRS has received and processed over 300,000 incident 
reports, including over 30,000 in 1997 alone. Nearly 
70,000 reports are fully documented in the ASRS 
database. Reports are typically submitted by flight crews, 
but there are also many submissions by air traffic 
controllers, cabin crews, ground crews, and others. Each 
report in the database includes a narrative section, as well 
as specialized data fields to aid in classification and 
statistical analyses. Many narrative sections include only 
the original narrative received from the incident reporter, 
but some contain additional material. This can include 
supplemental information obtained from the incident 
reporter through "callback conversations," as well as 
original narratives from others reporting the same 
incident. Figure 1 shows a typical incident report. 

QUORUM text processing 

QUORUM is a collection of NASA-developed methods 
and software for analyzing, modeling, and relevance- 
ranking text such as the narratives in the ASRS database. 
QUORUM models text as a list of word pairs, each 
accompanied by a measure of their contextual relatedness, 
the proximky-weighted co-occurrence metric. Taken 
together, these pair-wise word relations constitute a 
weighted network model of the text. QUORUM 
relevance-r inks text by comparing QUORUM models. 

In a recent study (McGreevy, 1997), QUORUM was used 
to relevance-rank sentences from a collection of news 
stories describing the crash of TWA Flight 800, as well as 
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narratives and sentences from various collections of 
ASRS narratives. The criteria of relevance included 
typicality within a collection, typicality with respect to 
text outside the collection, association with one or more 
topics, and similarity to example narratives. For example, 
QUORUM was used to rank sentences from ASRS 
narratives according to their relevance to the news stories, 
and to rank the narratives in a collection according to 
their similarity to particular narratives of interest. In 
addition to modeling for relevance-ranking, QUORUM 
modeling has been used for fine-grained analysis of a 
collection of mode-related ASRS narratives (McGreevy, 
1996). Earlier work, in which the QUORUM method was 
initially developed, involved modeling of scientific text 
describing satellite-based volcanology (McGreevy, 1995). 

In previous studies, QUORUM was applied to collections 
containing up to hundreds of ASRS narratives. In the 
present study, QUORUM was applied to all of the 
narratives in the ASRS database that were available at the 
start of the study, a collection of 67820 narratives. In 
earlier studies, QUORUM relevance-ranked hundreds of 
ASRS narratives based on a few other narratives or on 
approximately 100 brief news accounts of an aviation 
disaster. In the present study, QUORUM was used to rank 
all of the narratives of the ASRS database according to 
their relevance to two official government accident 
reports. Finally, this is the first QUORUM study in which 
ASRS analysts have rated the results. 

Design of the study 

This was an informal, exploratory study, designed to 
obtain the first relevance ratings of QUORUM - 
selected narratives by qualified ASRS analysts. The 
study was primarily intended to see if a useful 
proportion of relevant incidents might be retrieved by 
using an accident report as the source of the query 
model. The study was further intended to provide 
some insight into the nature of the operational topics 
contained in the relevant incidents, and whether those 
topics might have a bearing on the prevention of 
future accidents. 

From the point of view of improving QUORUM'S 
performance, the study was intended to provide 
insight into the similarities and differences between 
QUORUM’S ratings of relevance and those of 
analysts. The use of six different assertions of 
relevance was intended to provide information 
regarding the kinds of relevance found among 
QUORUM-selected reports. This is of particular 
interest to those seeking incidents having relevance to 
the causes of the Cali accident. 


In addition to the informal, exploratory questions 
above, the study also involved the experimental 
question of whether QUORUM performs significantly 
better than chance. Thus, as a control, 100 randomly 
selected narratives were also rated by the analysts. 

This provides one estimate of the distribution in the 
ASRS database of narratives that are relevant to the 
Cali accident. Without the control, it would be 
difficult to interpret QUORUM’S performance. The 
tested hypothesis was: The mean relevance rating of 
incidents in the collection of QUORUM-selected 
incidents is significantly higher than the estimated 
mean relevance rating of the 67820 incidents from the 
ASRS database. 

Method of the study 

Two official government documents that describe and 
analyze the Cali accident were used to rank 67820 
incident narratives from the Aviation Safety Reporting 
System (ASRS) database according to their relevance to 
the Cali accident. The 100 narratives having the highest 
relevance-ranking values were selected for inclusion in 
the study. An additional 100 incident narratives were 
selected at random from among the 67820 narratives. 

None of the narratives in the random collection were also 
among the 100 narratives of the relevance-ranked 
collection. Three professional incident analysts from the 
ASRS read the Cali documents, and read and rated each 
of the 200 incident narratives according to their relevance 
to the Cali accident. The analysts had no indication of 
QUORUM'S ratings, nor did they know that half the 
incidents had been selected randomly. 

Analysts 

Each of the analysts has had extensive experience in 
commercial aviation operations. The analysts include: 

• a retired air carrier pilot with 50 years of aviation 

experience including 8 in the U. S. Air Force, 33 with 
a major U. S. air carrier, and 9 as an ASRS analyst; 

• a retired air traffic controller with 33 years experience 

at tower, TRACON, and center facilities and 6 
months experience as an analyst at the ASRS, and; 

• a retired corporate pilot with 22 years of flying 

experience and 9 years experience as an analyst and 
research scientist at ASRS. 

Procedure 

Modeling the Cali documents — QUORUM was 
used to model the Cali documents and to generate a 
query model based on those documents. The method 
is described in detail in appendix 1 . QUORUM 
methods are described in even greater detail in 
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McGreevy (1997), McGreevy (1996), and McGreevy 
(1995). In modeling text, QUORUM generates lists of 
word pairs that are often found in close proximity in 
the text. For each word pair, it calculates a proximity- 
weighted co-occurrence metric. This metric indicates 
the tendency of the two words to be found in the same 
context. A word pair and the corresponding relational 
metric value (RMV) is called a QUORUM relation. 
The initial model of the Cali documents consisted of 
5000 QUORUM relations, while the query model 
consisted of 2436 QUORUM relations (table 1). So- 
called "stop words” (e.g., the, is, was, and, or, though) 
were not included in the model. Because all text in the 
ASRS narratives (and their models) is in uppercase 
format, the Cali documents were capitalized prior to 
modeling. All of the text in the Cali documents was 
used, including not only the body of the text, but also 
such text as the table of contents, headers, tables, lists, 
and references. 

The initial model of the Cali documents could be used 
directly as a query model, but this has two major 
drawbacks. First, the vocabulary of the Cali 
documents does not match the abbreviations of the 
ASRS narratives. Second, use of the document model 
tends to retrieve incidents that are relevant in a highly 
generic way. In particular, the incidents tend to 
involve the approach phase, but without emphasis on 
the aspects of the Cali accident that make it unusual. 

To eliminate the vocabulary mismatch problem, the 
prominent Cali vocabulary was mapped to the ASRS 
database abbreviations prior to modeling. For 
example, the word pair "first officer" was mapped to 
”FO” and the word "approach" was mapped to 
"APCH". These mappings would be unnecessary if a 
collection of ASRS incident reports were used to 
generate the query model because the vocabulary 
would be identical. 

To eliminate the problem of retrieving generic 
incidents, the initial model of the Cali documents was 
transformed. The relational metric value of each word 
pair in the initial document model was divided by the 
relational metric value of that word pair in all 67820 
narratives from the ASRS database. This tends to 
disfavor word pairs such as APCH RWY (approach 
and runway) since they are so common in the ASRS 
database. Notice, however, that such generic word 
pairs are not eliminated, just devalued. The weighting 
favors word pairs that often share the same contexts in 
the Cali documents, but only rarely do so in the ASRS 
database. For example, the word pair TERRAIN 
CALI is exceedingly rare in the ASRS database, but 


prominent in the Cali documents, so it gains 
prominence. 

If a pair of w ords is never present, proximal, or 
sufficiently proximal, then it falls out of the query 
model. For example, the word pair "Civil 
Aeronautica" is very prominent in the Cali documents, 
but never occurs in the ASRS database, so it falls out 
of the query model. The point at which occurring 
word pairs fall out of the model depends on the 
resolution of the document models. A single 
occurrence cf a word pair is included in the initial 
model of the Cali documents if the two words are 
separated by no more than 12 other words. This is 
more than sufficient resolution for modeling the text. 

The method of favoring certain co-occurrences is, by 
itself, somewhat brittle since even marginally 
important word pairs from the Cali documents can 
appear to be very important in the query if they are 
very rare, but present, in the ASRS database. To adjust 
for this, the relational metric value associated with 
each word pair is also multiplied by the frequencies of 
those words in the Cali documents. Thus, word pairs 
in the query model are favored if each of the words is 
important in the Cali documents, if the word pairs are 
often found n the same contexts in the Cali 
documents, and if they are rarely found together in the 
ASRS database. 

There remains an effect of "out-of-domain" 
vocabulary, which was not addressed in this study. 
This is a problem in which words such as "accident," 
which are common in the Cali documents, are given 
excessive importance due to their rarity in the ASRS 
database. This problem could have been addressed, 
but it would have required manual tuning of the 
models. While one of the strengths of QUORUM is 
that its models are fully accessible and finely tunable, 
it was decided that, for the sake of this exercise, it 
would be better only to use methods that can be 
applied with little or no human intervention. Fine- 
tuning is addressed in more detail in the discussion 
section. 

Modeling the ASRS incident narratives — 

QUORUM was used to model each of the 67820 
ASRS incident narratives, producing 67820 models. In 
modeling each of the narratives, QUORUM generated 
lists of word pairs that are often found in close 
proximity in the text. For each word pair, a proximity- 
weighted co occurrence metric was computed. This 
metric indicates the tendency of the two words to be 
found in the same context. For this study, each model 
was limited to 100 word pairs and their associated 
relational metric values, that is, 100 QUORUM 
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relations. Thus, the 67820 narratives were modeled as 
6,782,000 QUORUM relations. Figure 2 shows an 
example of one of the narratives, and table 2 shows a 
QUORUM model of that narrative. 

As usual, the vocabulary of the ASRS narratives was 
standardized slightly for consistency. Such 
standardization is appropriate, though not strictly 
necessary, for all QUORUM-based processing of the 
ASRS database. It is only done during the modeling 
phase, and does not have any effect on the reports in 
the database. Examples of this standardization include 
mapping references to "F/O" (first officer) to "FO", 
and mapping a variety of references to various makes 
and models of aircraft to single representations. For 
example, ’’Boeing 757”, ”6-757”, ”B757”, and a 
variety of other forms were mapped to ”B757'\ In 
addition, some word pairs that function as units were 
linked, such as FLT_CREW, SPD_BRAKE, and 
ACR_X. 

It is possible for a particular pair of words to be 
present within the same narrative, and yet their degree 
of co-occurrence could be too small for the word pair 
to be included in the model. Since each narrative, 
whether long or short, was limited to 100 QUORUM 
relations for this study, the minimum resolution of the 
model varies from narrative to narrative. Given the 
average number of words in a narrative of 219 and a 
context window of ±20 words, the minimum relational 
metric value is typically around 19. This means that a 
single occurrence of a word pair in a narrative will 
appear in the model if the two words are immediately 
adjacent to each other. This is probably excessive 
resolution. Given the maximum word count of 1479 
and a context window of ±20 words, the minimum 
relational metric value is typically around 47. This 
means that two words would have to appear in close 
proximity (though not necessarily immediately 
adjacent) 3 or more times before being included in the 
model. If greater resolution is required, longer 
narratives could be modeled using more relations, in 
order to bring the minimum relational metric value to 
a lower value such as 19. 

Retrieving incident narratives — In order to find the 
ASRS incident narratives that are most relevant to the 
Cali accident, the Cali query model was compared 
with each of the 67820 narrative models to produce a 
relevance-ranking value for each narrative. The 
method is described in appendix 1 . The narratives 
with the 100 highest relevance-ranking values were 
selected for use in the study. 

To compare a query model and a narrative model, 
their intersection is found, producing an intersection 


model. The intersection model represents features of 
the narrative that are relevant to the Cali accident. The 
intersection model contains word pairs that appear in 
both the query model and the narrative model. For 
example, table 2 shows a narrative model in which 
word pairs that are also in the Cali query model are 
highlighted in bold italics. So, for example, since there 
is a relation in the query model that contains the words 
APCH (approach) and CHART, as well as a relation 
in the narrative model containing the same words, the 
word pair APCH CHART (or CHART APCH, since 
order doesn't matter in this situation) is included 
among the word pairs in the intersection model. 

The relational metric value (RMV) associated with 
each word pair of the intersection model is the product 
of the RMV in the query model and the RMV in the 
narrative model. For example, the word pair APCH 
CHART has an RMV of 277 in the query model (table 
1) and 67 in the narrative model (table 2) to produce 
an RMV of 18559 in the intersection model (table 3). 
Table 3 also shows the other relations of the 
intersection model for narrative 310130. 

In addition to an intersection model, table 3 also 
shows an example of how the relevance-ranking value 
(RRV) of a narrative is derived from its intersection 
model. Three factors are taken into consideration. The 
first factor is the sum of the weights (RMVs) of the 
intersection relations. This is a measure of the 
prominence in the Cali document and the prominence 
in the narrative. If this factor is high, then one or 
several elements of the incident are prominent in the 
Cali accident. The second factor is the fraction of the 
narrative model that intersects with the query model. 
This reflects the extent to which the commonalities of 
the incident and the Cali accident are central to the 
incident. Since this was found to favor short 
narratives, a third factor (the length factor) favors 
longer reports by multiplying by the number of words 
in the narrative, divided by the largest likely number 
of words. 

The product of these three factors, the relevance- 
ranking value (RRV), was calculated for each of the 
67820 narratives. The 100 narratives having the 
largest RRVs were collected for use in this study. 

There is a simpler, but more abstract way to describe 
the calculation of the relevance-ranking value (RRV). 
That is, if the query model and the narrative models 
are considered to be vectors, then an RRV consists of 
the inner product of the query vector and a narrative 
vector, multiplied by the fraction-of-intersection factor 
and the length factor. 
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An unused, but possibly beneficial scale factor is the 
fraction of the query model that intersects the 
narrative model. This could supplement the combined 
weights of the intersection relations by further 
reflecting the extent to which the commonalities of the 
incident and the Cali accident are central to the 
accident. 

For the sake of comparison, another 100 incident 
narratives were randomly selected from the database. 

This is useful because it is not known how many 
incidents in the ASRS database are relevant to the Cali 
accident. If the database were dedicated to Cali-like 
incidents (which it is not) then even a random 
collection of narratives would contain a large fraction 
of relevant reports. On the other hand, it might be that 
only a few reports are relevant to Cali. Without some 
idea of the number of relevant reports in the database, 
it would be difficult to interpret QUORUM'S ability to 
find the most relevant reports. The proportion of 
relevant narratives among the randomly selected 
narratives is an estimate of the proportion of relevant 
narratives in the database. By comparing the analysts' 
ratings of the random collection with their ratings of 
the QUORUM-selected collection, it is possible to see 
if QUORUM performed any better than chance. 

The order of the 200 incident narratives was 
randomized, and the narratives were printed and 
bound. With generous inter-line spacing to ease 
reading, the collection filled 141 pages. One copy was 
provided to each analyst. 

Rating by ASRS analysts — The three analysts each 
read the Cali documents as their schedules permitted. 
They then participated in a one hour brainstorming 
meeting to discuss among themselves the diversity of 
issues raised by the documents. The analysts then read 
the 200 narratives and rated their relevance, as their 
schedules permitted. Their instructions are shown in 
appendix 2, figure 1 . A sample page from the bound 
book of narratives is shown in appendix 2, figure 2. 

For each narrative, the analysts responded to six 
assertions of relevance: 

A) In some ways, the context of this incident is similar to 
the context of the Cali accident. 

B) Some of the events of this incident are similar to some 
of the events of the Cali accident. 

C) Some of the problems of this incident are similar to 
some of the problems of the Cali accident. 

D) Some of the human factors of this incident are similar 
to some of the human factors of the Cali accident. 


E) Some of the causes of this incident are similar to some 
of the cause s of the Cali accident. 

F) In some ways, this incident is relevant to the Cali 
accident. 

The purpose of the multiple statements was to assert 
relevance in a number of familiar and operationally 
useful ways. Specifically, the first five statements 
assert various aspects of relevance without using the 
word "relevance," and the last statement asserts 
relevance directly without specifying what aspect of 
relevance is involved. 

The assertions refer to six factors of relevance: 
context, eve nts, problems, human factors, causes, and 
one or more unspecified factors. That is, they refer to 
six kinds of relevance, the last of which is unspecified. 
The degree of orthogonality of these factors (the 
extent of their conceptual independence) is not at 
issue. Further, it is not required that any particular 
incident have multiple factors of relevance in order to 
be considered relevant. Since QUORUM makes no 
claim to detect any particular kind of relevance, any 
kind of relevance is acceptable. Accordingly, if the 
incident is relevant in any way, it is considered to be 
relevant. 

This breakdown of kinds of relevance, however, does 
allow the specific nature of the relevance to be 
reviewed. This is especially useful because causal 
relevance is of particular interest to investigators of 
accidents and incidents. If QUORUM found incidents 
having only contextual relevance but no other kinds of 
relevance, it would have limited practical value. The 
use of multiple assertions of relevance allows a review 
of this aspect of QUORUM'S performance. 

For each statement, the analysts selected from among 
seven responses: 

1) strongly disagree 

2) disagree 

3) somewhat disagree 

4) undecided 

5) somewhat agree 

6) agree 

7) strongly agree 

The essential purpose of the responses was to see 
whether the analysts rated the narratives as relevant or 
irrelevant. Thus, a rating of 5, 6, or 7 indicates a rating 
of "relevant," while a rating of 1, 2, or 3 indicates a 
rating of "irrelevant." 

Finding the extent to which the analysts agreed with 
the assertions of relevance was a secondary goal. By 
having these levels of agreement, it is possible to gain 
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insight into the strength of analysts' opinions that 
incidents are relevant and which narratives are 
considered to be the most relevant. 

The analysts used printed response forms to record 
their ratings. A sample page from the response form 
booklet is shown in appendix 2, figure 3. 

As an example of the resulting data, the ratings of 
incident narrative 310130 are shown in table 4. 

Results of the study 

The results of this study indicate that QUORUM 
successfully retrieved a large proportion of incidents that 
were judged to be relevant to the Cali accident. That is, 
QUORUM relevance-ranked 67820 incident narratives 
from the ASRS database, and the top 100 of these were 
judged to contain a large proportion of relevant incidents. 
Specifically, eighty-four incidents were rated by one or 
more of the analysts as relevant to the Cali accident. Each 
of the analysts rated nearly two-thirds of the QUORUM'S 
top 100 incidents as relevant to the Cali accident. 

The results in the following sections are presented from a 
variety of viewpoints. First, the ratings by each analyst 
are shown, comparing the proportion of relevant incidents 
in the QUORUM collection to that in the random 
collection. Next, the ratings of each of the six assertions 
of relevance are shown, indicating the proportion of 
incidents rated as relevant to the context, events, 
problems, human factors, causes, and unspecified factors 
of the Cali accident. Following this, the degree of 
relevance of the QUORUM collection is compared with 
that of the random collection. This is based on the number 
of assertions of relevance to which the analysts agreed, 
for each incident. 

Next, the strength of analysts' opinions is presented, based 
on the seven levels of agreement the analysts used in 
rating each narrative. These data are also used to sort the 
collections of incidents on relevance. 

Several aggregations of the data are also shown, including 
a comparison of the mean relevance ratings of the 
QUORUM collection and the random collection, a 
measure of the consistency of ratings among the six 
assertions of relevance, and correlations among the 
ratings of the analysts and between the ratings of the 
analysts and QUORUM. 

These various viewpoints indicate that QUORUM 
successfully retrieved a large proportion of incidents that 
are relevant to the Cali accident. This shows that a 
QUORUM-derived query model based on accident 
reports can be used with high precision to retrieve 


relevant incident reports. The results also indicate that 
QUORUM performed significantly better than chance. 

In addition, the results indicate that the ratings among the 
analysts, and between QUORUM and the analysts, were 
consistent, though not identical. The issue of differences 
of opinion among the analysts, and between the analysts 
and QUORUM, is addressed in the discussion section 
The discussion section also includes an outline of the 
operational topics contained in the relevant incidents 
selected by QUORUM. These topics clearly have a 
bearing on the prevention of future accidents. 

(Note: References to analysts by number, such as "analyst 
1", are unrelated to the order that the analysts are listed in 
the method section, "Analysts.") 

Incidents rated relevant by each analyst 

The results indicate that each of the analysts rated a 
large proportion of the QUORUM-selected incident 
narratives as relevant to the Cali accident. Each 
analyst found that approximately two-thirds of the 
QUORUM-selected narratives are relevant, compared 
with approximately one quarter of the randomly 
selected incidents (figure 3). Specifically, figure 3 
shows that analyst 1 found 63 of 100 QUORUM- 
selected narratives to be relevant, compared with only 
24 of 100 randomly selected narratives. Similarly, 
analyst 2 found 71 of 100 QUORUM-selected 
narratives to be relevant, compared with only 23 of 
100 randomly selected narratives. Further, analyst 3 
found 70 of 100 QUORUM-selected narratives to be 
relevant, compared with only 19 of 100 randomly 
selected narratives. 

Incidents rated relevant to each factor 

Figure 4 shows that the analysts rated a large 
proportion of the incident narratives in the QUORUM 
collection as relevant to the context, events, problems, 
human factors, causes, and unspecified factors of the 
Cali accident. Seventy of 100 QUORUM-selected 
narratives were rated as relevant to the context of the 
Cali accident by one or more of the three analysts. 
Similarly, seventy of 100 narratives were rated as 
relevant to events of the accident, seventy-four of 100 
narratives were rated as relevant to problems of the 
Cali accident, and 67 of 100 narratives were rated as 
relevant to human factors of the accident. While 
QUORUM is not designed to detect any particular 
kind of relevance, it is interesting to note that 54 of the 
100 QUORUM-selected narratives were rated as 
causally relevant. Causal relevance is a stringent 
criterion, yet over half the narratives QUORUM rated 
as relevant were rated by the analysts as causally 
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relevant. The analysts rated 61 narratives as relevant 
to unspecified factors of the Cali accident. 

By comparison, figure 5 shows that few narratives in 
the random collection were rated as relevant to the 
context, events, problems, human factors, causes, or 
unspecified factors of the Cali accident. The analysts 
rated 10 of 100 randomly selected narratives as 
relevant to the context of the Cali accident, and 19 of 
100 as relevant to events of the accident. They rated 
23 of 100 narratives as relevant to problems of the 
Cali accident, and 26 of 100 narratives as relevant to 
human factors of the accident. The analysts rated 14 of 
100 narratives as relevant to causes of the Cali 
accident, and 5 of 100 as relevant to unspecified 
factors. 

In figures 4 and 5, a narrative is counted as relevant if 
at least one of the analysts agreed to the corresponding 
assertion of relevance. For example, the pie chart in 
figure 4 labeled "Events" shows 75 relevant narratives 
because for each of 75 narratives, at least one of the 
analysts agreed with the statement: "Some of the 
events of this incident are similar to some of the 
events of the Cali accident." This approach to scoring 
is intended to reflect the expectation that each of the 
analysts bases his judgments of relevance on 
experiences, expertise, and insights that do not 
necessarily intersect with those of the other analysts. 

In addition, there is evidence that when analysts 
disagree among themselves about the presence of 
relevant factors, the analyst denying that presence 
overlooked relevant factors. This issue is explored in 
the discussion section in "Disagreements among 
analysts." 

Consensus among analysts 

The analysts did not always agree on which incidents 
are relevant. As shown in figure 6, all three analysts 
rated 48 of 100 QUORUM-selected incidents as 
relevant, compared with only 9 of the randomly 
selected incidents. At least two of three analysts rated 
72 of 100 QUORUM-selected narratives as relevant, 
compared with only 16 of 100 randomly selected 
narratives. At least one of three analysts rated 84 of 
100 QUORUM-selected narratives as relevant, 
compared with 41 of 100 randomly selected 
narratives. 

Whether rating QUORUM-selected incidents or 
randomly selected ones, the analysts agreed among 
themselves regarding the relevance of about two- 
thirds of the incidents. The pie chart on the left in 
figure 6 shows that 48 of 100 QUORUM-selected 
incidents were rated as relevant by all three analysts, 


and 16 of 100 were not rated as relevant by any of the 
three analysts, for a total of 64 unanimous ratings. In 
comparison, 36 of the QUORUM-selected incidents 
were rated as relevant by one or two of the analysts 
but not by the other analyses). Similarly, the pie chart 
on the right in figure 6 shows that 9 of 100 randomly 
selected incidents were rated as relevant by all three 
analysts, and 59 of 100 were not rated as relevant by 
any of the three analysts, for a total of 68 unanimous 
ratings. In comparison, 32 of 100 randomly selected 
incidents were rated as relevant by one or two of the 
analysts and not by the other analyst(s). 

Disagreements among the analysts, and some 
incidents about which they disagree, are reviewed in 
more detail n the discussion section in 
"Disagreements among analysts." 

Degree of relevance of the collections 

If a collection contains a large proportion of incident 
narratives that are highly relevant, then that collection 
is itself highly relevant. One measure of the degree of 
relevance of a particular narrative is the number of 
assertions of relevance to which one or more analysts 
agree. For example, if an analyst agrees with one 
assertion of relevance, it would indicate that the 
narrative is relevant, while agreement with two 
assertions of relevance would indicate that the 
narrative ha> a higher degree of relevance. Agreement 
with six assertions of relevance would indicate a very 
high degree of relevance. Thus, to gain insight into the 
degree of relevance of a collection of incidents, it is 
useful to sec the number of narratives that are relevant 
to a particular number of assertions of relevance. 
Accordingl) , figure 7 shows the number of incident 
narratives rated as relevant to the Cali accident in 
response to N of six assertions of relevance, for values 
of N ranging from 6 to 1 . 

The two pie charts of figure 7 show that the analysts 
rated many of the QUORUM-selected incidents as 
highly relevant, while few of the randomly selected 
incidents were rated as highly relevant. Thus, the 
QUORUM collection has a high degree of relevance, 
while the raidom collection has a low degree of 
relevance, as indicated by the following observations. 
In rating 45 of the 100 QUORUM-selected incidents, 
each of the six assertions of relevance was agreed to 
by one or more of the analysts. That is, 45 of the 100 
QUORUM-selected incidents were judged to be 
relevant to the context, events, problems, human 
factors, causes, and unspecified factors of the Cali 
accident. In comparison, only 3 of the 100 randomly 
selected incidents were judged to have all six 
relevance factors. Twelve of the 100 QUORUM - 
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selected incidents were judged to have exactly five of 
the relevance factors, so a total of fifty-seven of 100 
QUORUM- selected incidents were judged to have at 
least 5 of the relevance factors. In comparison, only 5 
of the 100 randomly selected incidents were similarly 
judged. Nine of the 100 QUORUM-selected incidents 
were judged to have exactly 4 of the relevance factors, 
so a total of 66 of 100 QUORUM-selected incidents 
were judged to have at least 4 of the relevance factors. 

In comparison, only 8 of the 100 randomly selected 
incidents were similarly judged. Six of the 100 
QUORUM-selected incidents were judged to have 
exactly 3 of the relevance factors, so a total of 72 of 
the 100 QUORUM-selected incidents were judged to 
have at least 3 of the relevance factors. In comparison, 
only 16 of the 100 randomly selected incidents were 
similarly judged. Five of the 100 QUORUM-selected 
incidents were judged to have exactly 2 of the 
relevance factors, so a total of 77 of the 100 
QUORUM-selected incidents were judged to have at 
least 2 of the relevance factors. In comparison, only 
24 of the 100 randomly selected incidents were 
similarly judged. 

Finally, seven of the 100 QUORUM-selected 
incidents were judged to have exactly one of the 
relevance factors, so a total of 84 of the 100 
QUORUM-selected incidents were judged to have at 
least one of the relevance factors. In comparison, only 
41 of the 100 randomly selected incidents were 
similarly judged. Thus, 84 of the 100 QUORUM- 
selected incidents were judged to be relevant to the 
context, events, problems, human factors, causes, 
and/or unspecified factors of the Cali accident. 

Taken together, these results indicate that the 
QUORUM collection has a high degree of relevance, 
while the random collection has a low degree of 
relevance. 

Strength of analysts’ opinions 

In responding to each of the six assertions of relevance, 
the analysts varied in their level of agreement or 
disagreement. This was captured by the 7-level response 
scale shown in the section, '’Rating by ASRS analysts.” 
So, for example, if the analysts agreed with the assertion 
that "Some of the human factors of this incident are 
similar to some of the human factors of the Cali 
accident," they could "somewhat agree,” "agree,” or 
"strongly agree.” If they disagreed with that assertion, 
they could "somewhat disagree,” "disagree," or "strongly 
disagree." They were also free to declare that they were 
"undecided." 


In rating the random collection of 100 incidents, each 
analyst provided, in effect, his estimate of the number of 
incidents likely to be found at each level of agreement in 
any random sample of 100 incidents. In rating the 
QUORUM collection of 100 incidents, each analyst 
specified how many incidents at each level were actually 
retrieved by QUORUM. For example, regarding the level 
of agreement "strongly agree," the three pie charts on the 
right in figure 8 show that one is unlikely to find any 
incidents among a random collection of 100 incidents 
about which the analysts would "strongly agree" with an 
assertion of relevance. In contrast, the three pie charts on 
the left in figure 8 show that the analysts "strongly 
agreed" that 4 to 15 incidents in the QUORUM collection 
are relevant to the Cali accident. Here are the details. 
Analyst 1 did not "strongly agree" that any of the 
incidents in the random collection are relevant to the Cali 
accident, but "strongly agreed" that 4 of the QUORUM 
collection are relevant. Similarly, analyst 2 did not 
"strongly agree" that any of the random incidents are 
relevant, but "strongly agreed" that 12 of the QUORUM 
collection are relevant. Finally, analyst 3 "strongly 
agreed" that 1 incident from the random collection is 
relevant, but "strongly agreed" that 15 of the QUORUM 
collection are relevant. Thus, QUORUM performed much 
better than chance. 

Similarly, regarding the level of agreement "agree," the 
three pie charts on the right in figure 8 show that one is 
unlikely to find more than a few incidents, among a 
random collection of 100 incidents, about which the 
analysts would "agree" with an assertion of relevance. In 
contrast, the three pie charts on the left in figure 8 show 
that the analysts "agree" that 20 to 31 of the incidents in 
the QUORUM collection are relevant to the Cali accident. 
Here are the details. Analyst 1 "agreed" that 1 of the 
incidents in the random collection is relevant to the Cali 
accident, but "agreed" that 24 of the QUORUM collection 
are relevant. Similarly, analyst 2 "agreed" that 1 of the 
random incidents is relevant, but "agreed" that 31 of the 
QUORUM collection are relevant. Finally, analyst 3 
"agreed” that 4 incidents from the random collection are 
relevant, but "agreed” that 20 of the QUORUM collection 
are relevant. Again, QUORUM performed much better 
than chance. 

Each of the remaining levels can be interpreted in a 
similar fashion. This analysis leads to the conclusion that 
the analysts more strongly agreed that the QUORUM- 
selected incidents, rather than the randomly selected ones, 
are relevant to the Cali accident. Further, the analysts 
more strongly disagreed that the randomly selected 
incidents are relevant. 
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Appendix 3 presents the data in a different way. It uses 
the strength-of-opinion data to sort the incidents 
according to relevance. In appendix 3, table 1, all 100 
QUORUM-selected incident narratives are shown sorted 
in order of their ratings by the analysts. In appendix 3, 
table 2, all 100 randomly selected incident narratives are 
shown sorted in order of their ratings by the analysts. 

While there are several useful ways to sort these tables on 
relevance, the method that was used sorts on the number 
of scores at each level of agreement. Thus, if all three 
analysts strongly agreed with any of the assertions of 
relevance for a particular incident, then that incident is 
considered to be more relevant than one for which only 
two analysts strongly agreed with assertions of relevance. 
Further, if a single analyst strongly agrees that an incident 
is relevant, that incident is ranked as more relevant than 
one in which all three analysts merely agreed, but none 
agreed strongly. A review of the narratives themselves 
suggests that this sorting method produces a list that is 
reasonably well- sorted on degree of relevance. 

In figure 8 and in appendix 3, each analyst's highest rating 
across the six assertions of relevance is used to 
characterize each narrative because an agreement with 
any assertion of relevance is an acknowledgment that the 
narrative has some kind of relevance. For example, if an 
analyst "strongly agrees" with the assertion that the 
human factors of an incident are relevant to the Cali 
accident, but disagrees that there are other similarities, it 
remains true that the analyst "strongly agrees" that the 
narrative has relevance to the Cali accident. 

The significance of QUORUM'S performance 

A simple test was conducted to investigate the 
significance of the difference between the estimated mean 
relevance rating of the 67820 incident narratives from the 
ASRS database, and the mean relevance rating of the 
collection of QUORUM-selected incident narratives. The 
mean rating of the collection of randomly selected 
incidents was used as an estimate of the mean relevance 
rating of the ASRS database. The hypothesis being tested 
was that the mean relevance rating of incidents in the 
collection of QUORUM-selected incidents is significantly 
higher than the estimated mean relevance rating of the 
67820 incidents from the ASRS database. If true, this 
would indicate that QUORUM had performed 
significantly better than chance. 

One rating was used to represent each incident narrative: 
the highest rating across response statements and analysts. 
This rating indicates whether any of the analysts judged 
the incident to be relevant in any of the six ways (context, 
events, problems, human factors, causes, or unspecified 
relevance). 


The mean rating of the QUORUM collection, averaged 
across narratives, is 5.40. The mean rating of the random 
collection, averaged across narratives, is 3.61. 

Based on the ratings, the test statistic t (Kanji, 1993, pg. 
27) is equal to 13.275. Comparing this value with 
Student's t distribution with 99 degrees of freedom 
(Snedecor and Cochran, 1989, pg. 466), the probability of 
achieving a value of this magnitude is less than 0.001. 
That is, there is less than 1 chance in 1000 that the mean 
rating of QUORUM’S collection could have been 
achieved by random sampling. 

Thus, the mean rating of incidents in the collection of 
QUORUM-selected incidents is significantly higher than 
the estimated mean rating of the 67820 incidents from the 
ASRS database. Such performance is very unlikely to 
have been achieved by chance. 

Consistency among the assertions of relevance 

Cronbach's alpha, a measure of the internal consistency of 
a collection of response statements (Spector, 1992), was 
used to measure the consistency among the six assertions 
of relevance. Cronbach's alpha measures how well the 
statements reflect a common, underlying construct. A 
value of 0.7 or greater is considered to be an indication of 
internal consistency. 

Based on the analysts’ ratings of the 200 narratives, 
Cronbach’s alpha is 0.962, indicating that responses to the 
six assertions of relevance are highly intercorrelated, and 
that the responses to the individual statements reflect 
responses to a common, underlying construct. 

Cronbach’s alpha does not guarantee that the meaning of 
the common, underlying construct is the same as the 
intended meaning. That meaning is carried by the 
wording of the assertions, and must be interpreted by the 
analysts. 

The purpose of the multiple statements was to assert 
relevance in a number of familiar and operationally 
useful ways Specifically, the first five statements 
assert various aspects of relevance without using the 
word "relevance," and the last statement asserts 
relevance directly without specifying what aspect of 
relevance is involved. The wording of the assertions 
centers on two concepts. The first concept is similarity 
between features of an incident and features of an 
accident. Th e second concept is the relevance of an 
incident to £n accident. Cronbach's alpha indicates that 
the statements based on these concepts elicited 
consistent responses from the analysts. The high value 
of Cronbach’s alpha indicates that, as intended, the 
various que: tions did address a single underlying 
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concept, which, in this case, could be called either 
feature similarity or relevance. 

While the intended meaning of the assertions is clear, the 
analysts' ratings show that the assertions measure what 
was intended. A reading of the incidents rated highest by 
the analysts (e.g., the 10 incidents in appendix 4) 
indicates that they are highly relevant to the Cali accident. 
Those rated lowest generally have little or no relevance. 
This is further evidence that the six assertions of 
relevance do, in fact, measure analyst's opinions of 
relevance. 

Correlations 

The broad pattern of agreement among the analysts, and 
between the analysts and QUORUM, can be seen by 
looking at the correlations among the ratings. Since any 
kind of relevance is of interest, the maximum rating 
across assertions of relevance is used to calculate the 
correlations reported here. Further, the correlations are 
based on the ratings of all 200 narratives. Given the 198 
degrees of freedom, these observed values of r, the 
correlation coefficient, are all statistically significant at 
the 1% level (Snedecor and Cochran, 1989, pg. 473). 


X 

Y 

r 

analyst 2 

analyst 3 

0.706 

analyst 1 

analyst 2 

0.616 

analyst 1 

analyst 3 

0.573 

QUORUM 

analyst 3 

0.590 

QUORUM 

analyst 2 

0.560 

QUORUM 

analyst 1 

0.489 


Among the analysts, the ratings of analysts 2 and 3 are 
most correlated, while the ratings of analysts 1 and 3 are 
least correlated. QUORUM’S ratings are most correlated 
with those of analyst 3, while QUORUM'S ratings are 
least correlated with those of analyst 1 . It is interesting to 
note that QUORUM'S ratings are more correlated with the 
ratings of analyst 3 (r=0.590) than the ratings of analyst 1 
are correlated with the ratings of analyst 3 (r=0.573). This 
suggests that QUORUM'S ratings are worthy of 
comparison with those of human raters. 

Discussion 

The results indicate that the analysts rated many of the 
QUORUM-selected incidents as relevant, and few of 
the randomly selected incidents as relevant; that 
QUORUM performs significantly better than chance; 
and that QUORUM can retrieve a large proportion of 
relevant incident reports based on analysis of accident 
reports. 

Still, the analysts did not always agree regarding the 
relevance of particular incident narratives. Whether 


rating QUORUM-selected incidents or randomly 
selected ones, the analysts agreed among themselves 
regarding the relevance of about two-thirds of the 
incidents. This issue is discussed in the section, 
"Disagreements among analysts." 

From the viewpoint of commercial aviation 
operations, it is useful to examine the nature of the 
incidents retrieved by QUORUM. Even though many 
of the incidents were rated as relevant, which of the 
important Cali-related factors do they contain? Are 
they all "controlled flight toward terrain" (CFTT) 
incidents, or are other factors involved? The section 
"Prominent factors among the relevant narratives" 
examines these questions. 

Finally, QUORUM was occasionally misled by the 
presence of words such as "accident," "FAA," 

"safety," and "civil." This highlights a particular 
concern when using accident reports to find incidents, 
as discussed in the section, "Improving QUORUM’S 
performance." 

Disagreements among analysts 

It was not expected that the three analysts would 
always agree in their opinions. This is particularly true 
because the analysts in this study came from three 
different roles within the domain of commercial 
aviation. One analyst was a pilot for a major 
commercial airline, another was a corporate pilot, 
while the third was an air traffic controller. While they 
all shared the same aviation environment, their 
particular roles were complementary. Thus, while 
these professionals share a certain amount of expertise 
and experience, much of their domain knowledge and 
insight is unique to each of them. As a result, one 
would not expect the three analysts to have identical 
interpretations of the Cali accident. Neither would one 
expect that the analysts would have identical 
appreciation of the relevant factors of every incident. 

Given this expected diversity of insight and opinion, 
care must be taken when combining analysts’ ratings. 
In figures 4 and 5, for example, a narrative is counted 
as relevant if at least one of the analysts agreed with 
each of the assertions of relevance. This approach to 
scoring is intended to reflect the expectation that each 
of the analysts bases his judgments of relevance on 
experiences, expertise, and insights that do not 
necessarily intersect with those of the other analysts. 

This approach to scoring also reflects the fact that 
there is evidence, examples of which are shown 
below, that when analysts disagree among themselves 
about the presence of relevant factors, the analyst 
denying that presence seems to have overlooked 
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relevant factors or perhaps made an error in marking 
the rating form. (It is important to note in this context 
that the tasks involved in this study are quite different 
from the tasks usually performed by ASRS analysts. 

In no way does this study have anything to say about 
their performance of ASRS tasks. It was not designed 
to investigate such issues.) 

The rating of incident narrative 310228 is an example 
in which analysts had diametrically opposite opinions. 
In rating that incident, analysts 2 and 3 either agree or 
strongly agree that the narrative is relevant to the 
context, events, problems, human factors, causes, or 
unspecified factors of the Cali accident, yet analyst 1 
disagrees or somewhat disagrees that any of the 
factors are relevant. A review of the narrative of 
incident 310228 (appendix 5, figure 1) suggests that 
analysts 2 and 3 are correct, while analyst 1 has 
apparently overlooked relevant factors. Just as at Cali, 
the incident involved a VOR/DME approach, there 
was miscommunication between the crew and the 
controller, the crew was confused about the approach, 
the flight was off course, and name confusion was a 
central factor. In fact, just as the Cali accident 
involved same-letter identifier confusion about a 
navigation fix (Rozo’s R vs. Romeo's R), this incident 
involved sound-alike name confusion about a 
navigation fix (Beeje vs. Meach). One must conclude 
that analyst 1 is mistaken in denying the presence of 
any relevant factors. 

In another example, in rating the incident narrative 
140711, analysts 1 and 3 either agree or somewhat 
agree that the narrative is relevant to the context, 
events, problems, human factors, causes, or 
unspecified factors of the Cali accident, yet analyst 2 
strongly disagrees that any of the factors are relevant. 
A review of the narrative of incident 140711 
(appendix 5, figure 2) suggests that analysts 1 and 3 
are correct, while analyst 2 seems to have overlooked 
relevant factors. The incident took place in Colombian 
airspace and involved one of the ATC facilities, 
Barranquilla Center, that was involved in the Cali 
accident. So, the context, at least, is nearly identical, 
not just similar. Just as in the Cali incident, the flight 
was cleared direct to the Tulua VOR. So, one of the 
events of the incident is identical, not just similar, to 
the Cali accident. A problem of the incident involved 
the quality, or perceived quality, of Colombian air 
traffic control, and this problem is highly relevant to 
the Cali accident. The performance of the air traffic 
controller is a human factors issue in both the incident 
and the Cali accident. In this incident, the direct 
clearance to the Tulua VOR, the performance of the 
Colombian controller, and the quality of Colombian 


ATC are, in some ways, similar to the causes of the 
Cali accider t. Given all of these similar factors, it is 
easy to see why analysts 1 and 3 rated the incident as 
relevant, and hard to see why analyst 2 did not. 

In yet another example, in rating incident narrative 
153355, analyst 1 agrees or somewhat agrees that the 
incident is relevant to the Cali accident. Analyst 2 
somewhat agrees or is undecided. Analyst 3, however, 
somewhat disagrees, disagrees, or strongly disagrees 
that the incident is relevant. A review of the incident 
(appendix 5. figure 3) indicates that it occurred during 
approach, it involved difficulty with the FMS and 
over-reliance on the FMS, and, even more relevant to 
the Cali accident, the crew was unsure of their 
position and the flight was off course. It appears that 
in rating this incident, analyst 3 overlooked relevant 
factors. 

As a final example, incident 355364 (appendix 5, figure 
4) was rated as irrelevant by analysts 1 and 3, while 
analyst 2 only somewhat agreed with one of the six 
assertions of relevance. In this case, only analyst 2 is 
correct. First, the incident and the Cali accident both 
occurred in the context of the approach phase of flight. 
More important, the incident involved two events that 
also occurred in the Cali accident. First, the crew received 
a GPWS (ground proximity warning system) warning. 
Second, the crew performed an escape maneuver. Only 
analyst 2 agreed with the statement, "Some of the events 
of this incident are similar to some of the events of the 
Cali accident." Analyst 1 flatly disagreed with this 
statement, and analyst 3 somewhat disagreed. Clearly, the 
GPWS warning and the escape maneuver were events that 
are similar lo those in the Cali accident. So analyst 1 and 
3 must have overlooked these events or perhaps 
misunderstood what is meant by "events." Or perhaps 
analysts 1 and 3 were expressing an opinion that nothing 
can be learred from this narrative that would have an 
influence on preventing future accidents like the one at 
Cali. If so, they failed to follow instructions. Besides, 
previous experiences with false GPWS warnings can be a 
factor in accidents involving controlled flight into terrain 
(Majikas, 1995). None of the analysts agreed with the 
statement, ’ In some ways, the context of this incident is 
similar to the context of the Cali accident." Apparently, 
the fact that this incident and the Cali accident happened 
during the approach phase was insufficiently important 
for the analysts to consider the contexts to be similar. 

These examples are typical of the cases in which the 
analysts disagreed among themselves regarding the 
relevance of particular narratives. Review of the ratings of 
these and o her similar cases indicates that when relevant 
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factors are present in an incident narrative, they are rarely 
overlooked by all three analysts. 

Prominent factors among the relevant narratives 

One might assume that all incidents related to a 
’’controlled flight into terrain" (CFIT) accident would be 
"controlled flight toward terrain" (CFTT) incidents. A 
review of the Cali accident, however, suggests that many 
other factors can play a role, as listed in the introduction 
of this paper (see section, "The Cali accident," on pg. 1). 
A review of the 84 QUORUM-selected incidents that the 
analysts rated as relevant to the Cali accident shows that 
the incidents contain a variety of similar factors, 
including over-reliance on automation, confusion during 
descent/approach, and operations in foreign airspace, as 
well as CFTT and GPWS (ground proximity warning 
system) alarms. 

The following outline shows the topical categories and 
subcategories of the 84 incident narratives that were rated 
relevant by QUORUM and one or more of the analysts. 
This outline is derived from the factors of the Cali 
accident listed on page 2 of this paper. The number in 
parentheses at the end of each line is the number of 
incidents in that category or subcategory. The numbers 
sum to 91 because Five of the incidents appear in two 
places, and another one appears in three places. Because 
of these and other cases of categorical overlap, the 
numbers in the headings should not be over-interpreted. 
(As used here, the term "automation" refers to the FMS 
(flight management system) or other components of the 
automated flight systems that are used to operate the 
aircraft.) 

1 . Over-reliance on automation, and other problems with 

use of automation (37) 

1.1. Over-reliance on automation (36) 

1.1.1. Automation turns aircraft off course (5) 

1.1.2. Loss of data when other data are entered (5) 

1.1.3. Distracted by automation (7) 

1.1.4. Name confusion using automation (4) 

1.1.5. Automation data entry error or data error (6) 

1.1.6. Other problems getting automation to work as 

desired (4) 

1.1.7. Miscellaneous over-reliance on automation (5) 

1 .2. Other automation-related problems ( 1 ) 

2. Confusion, changes, and other problems during 

descent/approach (29) 

2.1 . Last minute approach/runway change leads to 

significant confusion (3) 

2.2. Other confusion during descent/approach (11) 

2.2.1. Name confusion (5) 

2.2.2. Confusion regarding charts (4) 

2.2.3. Confusion due to use of wrong data (2) 

2.3. Forgot speed brakes (2) 


2.4. Other problems with changes late in 

descent/approach (4) 

2.5. Miscellaneous problems during descent/approach 

( 9 ) 

3. Terrain avoidance (19) 

3.1. GPWS alarms (15) 

3.1.1. Complacency, loss of situational awareness, slow 

reaction (5) 

3.1.2. Miscellaneous GPWS alarms (10) 

3.2. Other terrain-related incidents (4) 

4. Problems with operations in foreign airspace (6) 

4. 1 . Problems with operations in Latin America (4) 

4.2. Problems with operations in other foreign locations 

( 2 ) 

Appendix 6 shows excerpts of all 84 relevant narratives, 
organized according to the outline above. The outline and 
excerpts demonstrate that QUORUM-selected incidents 
contain a wide variety of the factors that are relevant to 
the Cali accident, with emphasis on some of the most 
important factors. Thus, many of the links of the chain of 
factors found in the Cali accident are found scattered 
among the incidents. Some incidents contain several of 
these factors, while others contain only one of them. 

Appendix 7 shows excerpts of all 16 of the narratives that 
were rated relevant by QUORUM but irrelevant by all 
three analysts. Review of these excerpts indicates that 5 
of these incidents are clearly irrelevant, 6 are vaguely 
relevant, and 5 are relevant. 

Taken together, appendices 6 and 7 contain excerpts of all 
100 of the incidents that QUORUM rated as being the 
most relevant to the Cali accident of the 67820 incidents 
from the ASRS database. 

Improving QUORUM'S performance 

QUORUM models can be refined by deleting relations 
that do not contain word pairs of interest. This is 
discussed in great detail in McGreevy (1997). While 
automatically generated query models can be very 
effective, as demonstrated by the results of this study, 
they can be improved by weeding out relations that are 
not of interest. Specifically, the number of QUORUM- 
selected incidents that are not relevant can be reduced by 
eliminating from the query model word pairs that are not 
operationally oriented, such as BELIEVES SAFETY. 

A typical sentence containing "believes” and "safety" 
from the accident reports is: 

"Therefore, the Safety Board believes that the 
FAA should require that all approach and 
navigation charts graphically present terrain 
information." 
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Because sentences like this are common in the documents 
used to generate the query model, word pairs like 
BELIEVES SAFETY appear to be important. They are, in 
fact, important, but only from the point of view of 
accident investigation and safety recommendations, not 
from the point of view of the operational factors of the 
accident. (In contrast, the word pair "TERRAIN 
CHARTS" from the same sentence is operationally 
oriented and would be retained.) The rarity of the word 
pair BELIEVES SAFETY in the ASRS database 
magnifies the apparent importance of this word pair. This 
can lead to retrieval of narratives that are not really 
relevant. The solution is to delete this and other similar 
word pairs from the query model because they come from 
the vocabulary of accident reports and safety 
recommendations, not from the vocabulary of operational 
problems. 

A further improvement can be obtained by removing 
words, word groups, and abbreviations that often appear 
in accident reports, but are not useful in identifying 
operational problems among incident narratives. 

Examples include: 

• International Civil Aviation Organization (ICAO) 

• National Transportation Safety Board (NTSB) 

• Federal Aviation Administration (FAA) 

• United States (U.S.) 

• investigation 

• accident 

• crash 

• time of impact 

The deletions would be applied to the accident documents 
before the query model is generated. (To avoid distorting 
the distances measured among other words, the deleted 
words would be replaced with generic non-words to act as 
place-holders.) 

Another class of words and word groups to be deleted 
includes those that are specific to a particular accident 
investigation. Among the most prominent of these terms 
in the Cali document are: 

• Aeronautica Civil 

• American Airlines 

• AA 

• Flight 965 

• AA965 

Words and word groups that are particular to the 
operational setting of the accident would not be deleted. 
These include, for example: 

• Tulua 

• Barranquilla Center 

• Colombia 

• Latin America 


Other ASRS incidents containing these words or word 
groups miglit be relevant to the operational problems 
associated with the Cali accident. 

A more radical idea, and perhaps a risky one, is to 
eliminate words that reflect an outsider's view of 
operations, since the core of ASRS narratives reflect an 
insider's vocabulary. Among these words are: 

• flight crew(s) 

• pilot(s > 

Members of flight crews more often refer to the captain or 
first officer. Review of irrelevant incidents retrieved by 
QUORUM {see appendix 7, sections 6 and 7) suggests 
that elimination of these words would reduce QUORUM’S 
false positive rate. 

In summary, QUORUM models can be easily refined by 
removing relations that are not of interest, but such 
refinement sometimes requires manually picking through 
the model relations. This effort is not necessary, as 
indicated b> QUORUM’S success in this study using an 
automatically generated query model. Still, it might be 
worthwhile in some applications to remove words, word 
groups, abbreviations, and word pairs that are oriented 
toward accident investigation and safety 
recommendations rather than operational details. Once 
the offending material is collected, it can be reused in 
subsequent analyses, allowing fully automated modeling 
and relevant e-ranking. 

Related work 

Using QUORUM for relevance-ranking puts it squarely 
in the technical domain of "information retrieval" (Frakes 
and Baeza- Yates, 1992). In that domain, nearly all work 
in relevance -ranking is based on the occurrence or 
frequency of words in documents (e.g., Salton, 1991), and 
very little of it is based on the co-occurrence of words. 
What co-occurrence work there is (e.g., Smadja, 1991) 
typically addresses analysis of word groupings such as 
"home run" rather than conceptual proximities and 
variably prcximal co-occurrences, and little of it directly 
involves inf ormation retrieval. 

Osgood (1959) pioneered the modeling of text based on 
conceptual (but not variably proximal) co-occurrences, 
but the method was not applied to information retrieval. 

Recently, Hawking and Thistlewaite (1996) have been 
developing ^-occurrence methods for information 
retrieval, but their methods are significantly different 
from QUORUM methods. For example, while 
QUORUM’:; query models are based on proximity- 
weighted co-occurrences among prominently occurring 
words in a body of text such as the Cali accident reports, 
Hawking and Thistlewaite base their query models on 
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unstructured and spontaneous generation of possibly 
occurring and possibly clustered words. 

A proximity method for information retrieval developed 
by Caid and Oing (1997) is based on the premise that if 
two different words are frequently found very close 
together in text, they are likely to be in similar categories, 
and thus have similar meanings or usages. Caid and Oing 
combine a document's word-categories into a single 
abstract category in order to summarize the contents of 
the document. Querying consists of finding abstract 
document categories that are similar to a particular 
abstract document category. In contrast, the QUORUM 
method is based on the premise that the associative 
structure of a text reflects the associative structure of the 
domain described in the text, as indicated by the concerns 
of the author (or authors). Concerns that are more 
frequently found in closer proximity in the text are those 
which are more strongly associated by the author. An 
explicit network of these associations constitutes a model 
of the text and a model of the concerns of the author. 
Querying consists of finding networks of concerns (e.g., 
models of ASRS narratives) that are similar to a particular 
network of concerns (e.g., a model of the Cali 
documents). 

Work related to QUORUM relevance- ranking is reviewed 
in McGreevy (1997). Work related to QUORUM 
modeling, upon which QUORUM relevance-ranking is 
based, is reviewed in great detail in McGreevy (1995). 

Conclusion 

QUORUM successfully retrieved a large proportion of 
incidents that are relevant to the Cali accident. Eighty - 
four of 100 QUORUM-selected narratives were rated as 
relevant to the Cali accident by one or more of the 
analysts. Each analyst rated approximately two-thirds of 
the QUORUM-selected narratives as relevant. Over two- 
thirds of the incidents retrieved by QUORUM were rated 
as relevant to the context, events, problems, or human 
factors of the Cali accident. Over half of the incidents 
were rated as relevant to the causes of the accident. 
Further, the QUORUM collection of incidents was 
significantly more relevant than the random collection. 
These results show that a QUORUM-derived query 
model based on accident reports can be used successfully 
to retrieve relevant incident reports. 

The ratings among the analysts, and between QUORUM 
and the analysts, were consistent, though not identical. 
Whether rating QUORUM-selected incidents or randomly 
selected ones, the analysts agreed among themselves 
regarding the relevance of about two-thirds of the 
incidents. This is attributed to the diversity of experience 
among the analysts, and to occasions in which one or two 


of the analysts recognize relevant factors that are 
overlooked by the other(s). As it turns out, QUORUM’S 
ratings are more correlated with the ratings of analyst 3 
than the ratings of analyst 1 are correlated with the ratings 
of analyst 3, suggesting that QUORUM'S ratings are 
worthy of comparison with those of human raters. 

The QUORUM-selected incidents contain a variety of 
factors that are similar to those of the Cali accident, 
including over-reliance on automation, confusion during 
descent/approach, and operations in foreign airspace, as 
well as CFTT and GPWS (ground proximity warning 
system) alarms. These topics clearly have a bearing on the 
prevention of future accidents. 
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Figures and Tables 




ACCESSION NUMBER : 204756 
DATE OF OCCURRENCE 
REPORTED BY 
PERSONS FUNCTIONS 
FLIGHT CONDITIONS 
REFERENCE FACILITY ID 
FACILITY STATE 
FACILITY TYPE 
FACILITY IDENTIFIER 
AIRCRAFT TYPE 
ANOMALY DESCRIPTIONS 

PROBLEM/LESS SEVERE; 


9203 

FLC; ; ; 

FLC , FO ; FLC , PIC . CAPT ; ARTCC , RDR ; 

VMC 
ONM 
NM 

ARTCC; 

ZAB; 

MLG; 

IN-FLT ENCOUNTER/ OTHER; ACFT EQUIPMENT 
ALT DEV/EXCURSION FROM ASSIGNED; NON 


ADHERENCE LEGAL RQMT/CLNC ; 

ANOMALY DETECTOR : COCKPIT / FLC ; 

ANOMALY RESOLUTION ; FLC OVERCAME EQUIP PROBLEM; FLC 

RETURNED ACFT TO ORIGINAL CLNC OR INTENDED COURSE ; 

ANOMALY CONSEQUENCES : NONE; 

SITUATION REPORT SUBJECTS : PROC OR POL ICY /COMPANY; AN ACFT TYPE; 

ACFT EQUIPMENT; 

NARRATIVE : AUTOPLT ON IN ' PERF 1 MODE, CRUISE 

CONDITIONS. ACFT STARTED A SLIGHT DSCNT TO ABOUT 300 FT BELOW 
ASSIGNED ALT, WHEREUPON CAPT SELECTED ’VERT SPD' MODE AND A 500 
FPM CLB . BUT ACFT STARTED TO CLB AT 2000 FPM AND WENT RIGHT 
THROUGH SELECTED ALT OF FL350 TO ABOUT 450 FT HIGH, WHEREUPON CAPT 
DISCONNECTED AUTOPLT AND RETURNED TO FL350. NO CONFLICT. I'M STILL 
NOT SURE IF THIS WAS DUE TO MOUNTAIN WAVE ACTIVITY OR AUTOPLT 


MALFUNCTION OR BOTH. CAPT ASSUMED MOUNTAIN WAVE AND INSTRUCTED ME 
TO RPT IT TO CTR. THIS PARTICULAR AUTOPLT, WHEN USED IN THE 'PERF 
CRZ ' MODE {WHICH IS SOP) CONSISTENTLY DEVIATES FROM SELECTED ALT 
BY + OR - 100 TO 200 FT. THIS MAKES IT AT TIMES DIFFICULT TO 
DETERMINE IF AUTOPLT IS FUNCTIONING 'NORMALLY' OR MALFUNCTIONING 
UNTIL IT IS TOO LATE. STILL, IF WE HAD BEEN MORE AGGRESSIVE IN 
DISCONNECTING AUTOPLT SOONER AND FLYING PROPER ALT, WE MIGHT HAVE 
DIMINISHED THE ALT EXCURSION. 

SYNOPSIS : CLR AIR TURB ASSOCIATED WITH MOUNTAIN 

WAVE ACTIVITY CREATES AN ALTDEV ALT EXCURSION. 

REFERENCE FACILITY ID : ONM 

FACILITY STATE : NM 

MSL ALTITUDE : 34700,35450 


Figure 1. Example incident report from the Aviation Safety Reporting System (ASRS) database . This report 
describes a situation involving an altitude deviation and the autopilot. 



Table 1. Excerpt of QUORUM query model derived from the Cali accident report documents , containing 2436 co- 
occurring word pairs. The top 100 relations are shown here , along with a sampling of less important relations in the 
model. Word pairs with larger relational metric values (RMVs) are the more important query relations. The query model 
was compared with each of the 67820 incident models in the database (e.g , Table 2), and for each incident , a single 
relevance-ranking value was calculated (for example, see Table 3). 


word 1 

word 2 

RMV 

CALI 

CLRED 

3643 

CALI 

TERRAIN 

35389 

PLTS 

PRESENT 

3637 

FMS 

PLTS 

25488 

FLT CREW 

RECOGNIZE 

3566 

FLT.CREW 

AWARENESS 

22072 

ACCIDENT 

B757 

3564 

CALI 

VOR 

18281 

NOT 

ICAO 

3510 

FMS 

FLT.CREW 

17511 

CTLR 

CIVIL 

3402 

FMS 

FAA 

17155 

PLTS 

SITUATIONAL 

3373 

ACCIDENT 

TERRAIN 

13275 

APCH 

INVESTIGATION 

3347 

TERRAIN 

CHARTS 

13131 

PLTS 

PROX 

3299 

FMS 

BOARD 

12754 

ACCIDENT 

TRAINING 

3188 

FAA 

CIVIL 

12533 

SPD.BRAKES 

CREW 

3166 

FMS 

TRAINING 

11469 

ACCIDENT 

CTLR 

3072 

TERRAIN 

FAA 

11355 

TRAINING 

CIVIL 

3063 

ACCIDENT 

OPS 

10890 

FMS 

STATES 

3034 

PLTS 

TERRAIN 

10210 

FLT.CREW 

PRESENT 

3011 

FLT.CREW 

TERRAIN 

10032 

ACCIDENT 

RTE 

2989 

TERRAIN 

AWARENESS 

10018 

CALI 

HIGH 

2979 

APCH 

ACCIDENT 

9999 

PLT 

SPD.BRAKES 

2967 

CALI 

DIRECT 

9314 

TERRAIN 

TRAINING 

2966 

APCH 

DISPLAYS 

8616 

APCH 

ROMEO 

2941 

FLT_CREW 

CRITICAL 

8417 

FLT CREW 

BELIEVES 

2883 

ACCIDENT 

CAPT 

8327 

FLT.CREW 

ACTIONS 

2845 

SAFETY 

BELIEVES 

7400 

FLT.CREW 

SITUATIONAL 

2822 

CAPT 

AMERICAN 

7230 

FMS 

INVESTIGATION 

2801 

TERRAIN 

USED 

6969 

ACCIDENT 

NAV 

2741 

FLT.CREW 

VOR 

6678 

CHARTS 

REQUIRE 

2723 

TERRAIN 

SIT 

6674 

FMS 

NDB 

2704 

PLTS 

CRM 

6525 

NOT 

AMERICAN 

2613 

TERRAIN 

SPD.BRAKES 

6439 

SAFETY 

BOARD 

2587 

APCH 

FLT.CREWS 

6301 

FAA 

EVALUATE 

2534 

APCH 

CIVIL 

6100 

SPD.BRAKES 

PITCH 

2534 

PLTS 

ACCIDENT 

6074 

OPS 

B757 

2513 

NAV 

REQUIRE 

6002 

SPD.BRAKES 

B757 

2498 

FMS 

COMMAND 

5990 

APCH 

FLT.CREW 

2493 

TERRAIN 

PRESENT 

5984 

FLT.CREW 

DECISION 

2464 

FMS 

PLT 

5934 

PLTS 

LATIN 

2459 

CAPT 

AA 

5760 

TERRAIN 

PATH 

2446 

BELIEVES 

BOARD 

5332 

FLT.CREW 

GPWS 

2441 

FMS 

CHART 

5258 

TERRAIN 

DATA 

2437 

APCH 

FMS 

5150 

PLTS 

AWARENESS 

2389 

ACCIDENT 

FO 

5038 

FMS 

CONFUSING 

2384 

CHARTS 

BELIEVES 

5007 

FMS 

CONFUSION 

2378 

FMS 

FIXES 

4719 

CIVIL 

INTL 

2327 

TERRAIN 

CFIT 

4550 

FLT.CREW 

MANEUVER 

2325 

FLT.CREW 

FLT.PATH 

4492 




VOR 

TULUA 

4491 

AWARENESS 

SIT 

1002 

FLT.CREW 

ESCAPE 

4489 




TERRAIN 

NAV 

4407 

POS 

WAYPOINTS 

278 

CALI 

COLOMBIA 

4032 

APCH 

CHART 

277 

ACCIDENT 

MIA 

3992 




NOT 

AA 

3909 

MANEUVER 

PITCH 

100 

TERRAIN 

REQUIRE 

3839 

NDB 

ENTERING 

100 

SPD BRAKES 

PERF 

3822 

... 



ACCIDENT 

ICAO 

3781 

SPD BRAKE 

LEVER 

10 

FMS 

ENVIRONMENT 

3717 

TERRAIN 

WARNINGS 

10 

FMS 

II 

3689 




CALI 

ATC 

3647 

WX 

CLOUDS 

1 
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310130 

WE WERE INITIALLY GIVEN A VECTOR DIRECT TO ARSOT AND PROGRESSIVE DSCNTS TO 5000 
FT. WE QUESTIONED THE CTLR REPEATEDLY ABOUT WHICH APCH WE COULD EXPECT. ATIS 
WAS GIVING ILS RWY 35 APCHS WITH A CIRCLE TO LAND ON RWY 11. THE WIND WAS 160 
DEGS AT 14 KTS, AND WE WERE NOT VERY HAPPY WITH THAT PROSPECT. WE HAD SCATTERED 
TO BROKEN CLOUDS AT ABOUT 1 300-1500 FT. I WAS HAVING A VERY DIFFICULT TIME 
UNDERSTANDING THE CTLR, AND THE FO AND SO WERE NOT DOING MUCH BETTER. THE ELS 
FOR RWY 1 1 WAS NOTAMED OTS. AT THE LAST MIN, AFTER WE WERE VECTORED DIRECT 
TOWARD THE OUTER LOCATOR 'OC', WE WERE CLRED FOR A 'STRAIGHT IN LNDG ON RWY 1 1' 

AND TOLD TO RPT OVER ’OC.' I HAD #1 VOR DME ON EZE AND THE FO INITIALLY SET UP 
HIS RADIO ON THE LOC 1 1 0. 1 , BUT THERE WAS NO LOC OR ANYTHING ON THAT FREQ. THE 
FO KEPT ASKING ME TO GET THE TYPE OF APCH AND ALT FROM THE CTLR. THE CTLR SAID 
TO FLY THE ALTS OF THE APCH. WE HAD BRIEFED BOTH THE ILS TO RWY 35 WITH A 
CIRCLE TO LAND AND THE LOC-VOR-DME RWY 1 1 APCH, BUT NOT A STRAIGHT IN APCH. THE 
ONLY STRAIGHT IN APCH WAS AN ADF LOCATOR APCH, WITH DME. OUR MINIMUM SECTOR ALT 
WAS 3000 FT AND WE WERE AT 5000 FT. WHEN WE WERE INSTRUCTED TO FLY THE ALTS ON 
THE APCH, FO (PF) SET THE ALT WINDOW TO 2150 FT WHICH WAS THE ALT FOR 10 DME ON 
THE 11-1 PAGE FOR A VOR APCH. MEANWHILE I WAS TRYING TO FIND AN APPROPRIATE 
APCH PAGE. WE SETTLED ON 1 1 -2 CHART SINCE THE CTLR HAD CALLED THE APCH A 
'STRAIGHT- IN APCH.’ THE SO WAS HELPING THE FO WITH HIS CHART AND RADIO SETUP, 

AND WE WERE DSNDING. I SAID 'I AM CONFUSED.’ I DIDNT UNDERSTAND WHY WE WERE 
DSNDING AND THE FO HAD ALL FLAGS WITH HIS RADIO ON THE ILS FREQ. I COULDN'T 
FIGURE OUT WHICH APCH HE WAS USING, AND I HAD TROUBLE READING HIS CHART FROM 
ACROSS THE COCKPIT. THEN THE SO MENTIONED THAT WE HAD A 3000 FT MSA. WE WERE AT 
2650 FT, AND I TOLD THE FO TO FLY AT 3150 FT WHICH WAS THE ALT FOR THIS POINT 
ON THE ADF LOCATOR APCH. HE CLBED BACK UP. WE GOT THE 2500 FT LIGHTS ON THE 
GPWS. I HAD SOME GND CONTACT INTERMITTENTLY, BUT I COULD NOT SEE THE RWY. THEN 
THE FO SWITCHED HIS RADIO OVER TO THE VOR FREQ 1 16.5 EZE AND CONTINUED THE 11-1 
APCH USING THE 1 1-2 PAGE. I THINK THE MAJOR PROBS WITH THIS APCH WERE: NO EARLY 
KNOWLEDGE OF WHICH RWY OR APCH WE WOULD USE. THE APCH WE WERE FINALLY GIVEN, OR 
FLEW ANYWAY, DID NOT CONFORM TO ANY OF THE PLATES. WE DID NOT MAINTAIN OUR MSA 
BTWN ARSOT AND 10 DME EZE. WE WERE CONFUSED ABOUT WHAT GND EQUIP WAS AVAILABLE 
TO US. I ACCEPTED THE CLRNC FOR A STRAIGHT-IN APCH, NOT KNOWING WHICH APCH. THE 
SOP WAS NOT FOLLOWED IN THAT THE PF (FO) RESET THE ALT WINDOW. I SHOULD HAVE 
GOTTEN CLARIFICATION OR REFUSED THE APCH UNTIL WE WERE SURE OF WHAT WE WERE 
DOING, INSTEAD WE FOUND NO ALT SHOWN FOR WHAT WE WERE DOING. 


Figure 2. Narrative of AS RS incident report number 310130, one of the 10 narratives rated as most relevant to the Cali 
accident. The QUORUM model of this narrative is shown in Table 2. A QUORUM model of the commonalities of this 
narrative and the Cali documents is shown in Table 3. The analysts' ratings of this narrative are shown in Table 4. 
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Table 2. A QUORUM model of the narrative of incident number 310130 (figure 2). Relations shown in bold italics are the 
word pairs that are also found in the query model (table 1) that was derived from the Cali documents. The sum of the 
relational metric values (RMVs) of the bold italicized relations is 1328. The sum of the other relations is 3185. So, the 
percentage of this model that is also found in the query model is 1 00‘(1 328/(1 328+31 85)) = 29.4261%. As shown in Table 
3, when the weights of the relations in this incident model are combined with the weights of the relations in the query 
model, the relation APCH CHART is seen to be the most important Cali-oriented relation in this narrative. The relations 
APCH NOT and APCH CTLR are the next most important Cali-oriented relations in this narrative. These three relations 
are underlined below. In general, relations of the form NOT X or X NOT suggest problems involving X. In this case, the 
relation APCH NOT suggests problems during the approach phase of flight. The relations APCH CHART and APCH 
CTLR suggest that the problems involved the approach chart and the approach controller. These indications are 
confirmed by reading the incident narrative in Figure 2. Similar difficulties were encountered just prior to the Cali accident. 


wordl 

word2 

RMV 

NOT 

BUT 

36 

APCH 

STRAIGHT 

186 

LAND 

CIRCLE 

36 

APCH 

m z 

135 

CTLR 

SAID 

36 

APCH 

ALT 

120 

APCH 

MAJOR 

36 

APCH 

RWY 

117 

APCH 

KNOWING 

36 

APCH 

CTLR 

105 

APCH 

LOC 

35 

APCH 

DME 

94 

APCH 

FREQ 

35 

APCH 

PAGE 

92 

VOR 

EZE 

34 

APCH 

FO 

88 

NOT 

DME 

34 

APCH 

VOR 

73 

APCH 

WINDOW 

34 

RWY 

ILS 

71 

NOT 

SOP 

33 

APCH 

ADF 

71 

NOT 

KNOWING 

33 

APCH 

LOCATOR 

70 

FO 

CTLR 

33 

APCH 

CHART 

67 

APCH 

MINIMUM 

33 

FO 

RADIO 

65 

VOR 

PAGE 

32 

RWY 

NOT 

61 

VOR 

OVER 

32 

FO 

ALT 

61 

FO 

DSNDING 

32 

RWY 

LAND 

59 

APCH 

TRYING 

32 

RWY 

CIRCLE 

59 

APCH 

THINK 

32 

NOT 

STRAIGHT 

58 

APCH 

SOP 

32 

FO 

NOT 

51 

APCH 

FINALLY 

32 

ALT 

WINDOW 

51 
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CTLR 

32 

DME 
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50 

WINDOW 

PF 

31 

APCH 

BUT 

49 

RWY 

APCHS 

31 

LOC 

BUT 

48 

NOT 

PLATES 

31 

RWY 

VOR 

47 

NOT 

FOLLOWED 

31 

APCH 

USING 

44 

NOT 

CONFORM 

31 

APCH 

KNOWLEDGE 

44 

FO 

WINDOW 

31 

APCH 

ILS 

44 

APCH 

RADIO 

31 

APCH 

EARLY 

44 

APCH 

MEANWHILE 

31 

STRAIGHT 

DME 

43 

LOC 

ANYTHING 

30 

APCH 

USE 

43 

FO 

BUT 

30 

APCH 

PF 

42 

APCH 

SECTOR 

30 

APCH 

ALTS 

42 

APCH 

LAND 

30 

ALT 

PF 

42 

APCH 

GIVEN 

30 

ALT 

DME 

42 

ALT 

DOING 

30 

RWY 

STRAIGHT 

41 

STRAIGHT 

BUT 

29 

FO 

FREQ 

41 

RADIO 

DSNDING 

29 

APCH 

CALLED 

40 

NOT 

MAINTAIN 

29 

APCH 

SETTLED 

39 

FO 

RWY 

29 

LOCATOR 

ADF 

38 

APCH 

CIRCLE 

29 

FO 

PF 

38 

ALT 

ALTS 

29 

FO 

LOC 

38 

SAID 

DSNDING 

28 

APCH 

PROBS 

38 

RWY 

LOC 

28 

RADIO 

FREQ 

37 

RADIO 

LOC 

28 

FO 

SET 

37 

ILS 

CIRCLE 

28 

DME 

EZE 

37 

DSNDING 

UNDERSTAND 

28 

APCH 

APPROPRIATE 

37 

DSNDING 

CONFUSED 

28 

STRAIGHT 

LOCATOR 

36 

DSNDING 

AM 

28 

RWY 

DME 

36 

CTLR 

ALTS 

28 

RWY 

BUT 

36 
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Table 3 . Derivation of the intersection model and the relevance ranking value (RRV) for the narrative of ASRS incident 
number 310130 (figure 2). The intersection model represents features of the narrative that are relevant to the Cali 
accident The intersection model is based on word pairs that appear in both the Cali query model (T able 1) and the 
incident model (Table 2). For example , the word pair APCH CHART appears in both the query model and the incident 
model. Accordingly ; the relational metric value (RMV) of APCH CHART in the query model (277, in column 3) is 
multiplied by the RMV in the incident model (67, in column 4) to produce the RMV of APCH CHART in the intersection 
model (18559, in column 5). The notes below the table show how the QUORUM relevance ranking value for this narrative 
is derived from this table, and how the QUORUM relevance rating is derived from the RRV. 


probe 

term 

term-in- 

context 

RMV in 
Cali query 

RMV in 
incident model 

product of 
RMVs 

APCH 

CHART 

277 

67 

18559 

APCH 

NOT 

33 

135 

4455 

APCH 

CTLR 

27 

105 

2835 

APCH 

DME 

21 

94 

1974 

APCH 

VOR 

22 

73 

1606 

APCH 

RWY 

13 

117 

1521 

APCH 

USE 

26 

43 

1118 

APCH 

FO 

12 

88 

1056 

APCH 

USING 

22 

44 

968 

RWY 

VOR 

18 

47 

846 

APCH 

RADIO 

14 

31 

434 

RWY 

DME 

12 

36 

432 

FO 

CTLR 

8 

33 

264 

APCH 

ALTS 

6 

42 

252 

APCH 

LAND 

7 

30 

210 

NOT 

DME 

6 

34 

204 

APCH 

BUT 

4 

49 

196 

DME 

VOR 

3 

50 

150 

RWY 

NOT 

2 

61 

122 

APCH 

ALT 

1 

120 

120 

FO 

RWY 

2 

29 

58 


To calculate the RRV: 

RRV = S * F * Wj / W m ax = 37380 * 0.294261 * 555 / 2000 = 3052 

where 

S = sum of products of RMVs (sum of last column) = 37380 

F = fraction of incident model that is matched by Cali model ; from table 2, F=29.426 1 % 

Wj = number of words in incident narrative 310130 = 555 

W m ax = a number larger than the number of words in the longest narrative in the database = 2000 

To convert QUORUM’S relevance ranking value (RRV) to QUORUM’S rating: 
rating = truncate(-0.75 + logio(RRV * (10 8 / max_RRV ))) 
if(rating>7)rating=7 
if(rating< 1 )rating= 1 

This calculation maps the RRVs into the same seven-level rating scale used by the analysts. 

From Appendix 3, Table 1, maxJRRV = 31696, so, given RRV of 3052, rating = 6 

Thus, in Appendix 3, Tablet, the QUORUM relevance ranking value (RRV) associated with ASRS incident 

number 310130 is 3052 and the QUORUM rating is 6. 
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Table 4 . Analysts' ratings of narrative 310130. The narrative is shown in figure 2. The analysts responded to each of the 
six assertions of relevance by selecting among the seven responses. 


report 

number 

analvst 

A: similar 
context 

B: similar 
events 

C: similar 
problems 

D: similar 
human factors 

E: similar 
causes 

F: unspecified 
relevance 

310130 

analyst 1 

6 

6 

7 

6 

5 

6 


analyst 2 

6 

6 

6 

7 

6 

6 


analyst 3 

7 

7 

7 

'7 

f 

7 

7 


A $$grt iQ P $ qL re lev a nce 

A) In some ways, the context of this incident is similar to the cor text of the Cali accident. 

B) Some of the events of this incident are similar to some of the events of the Cali accident. 

C) Some of the problems of this incident are similar to some of the problems of the Cali accident. 

D) Some of the human factors of this incident are similar to some of the human factors of the Cali accident. 

E) Some of the causes of this incident are similar to some of the causes of the Cali accident. 

F) In some ways, this incident is relevant to the Cali accident. 

Responses 

1 : strongly disagree 
2: disagree 

3: somewhat disagree 
4: undecided 
5: somewhat agree 
6: agree 

7: strongly agree 
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Number of relevant incidents identified by each analyst. 


OUORUM-selected incidents 
Analyst 1 



Randomly selected incidents 
Analyst 1 



Analyst 2 


Analyst 2 



Analyst 3 



Analyst 3 



Figure 3. Number of relevant incidents, among 100 QUORUM-selected incidents and 100 randomly selected 
incidents, identified by each analyst. 










Number of QUORUM-selected incidents that are relevant to 
the context, events, problems, human factors, causes, 
or unspecified factors of the Cali accident 


Context 


Human factors 




Events 


Causes 


undecided 




Problems 


Unspecified factors 




Figure 4. Number of QUORUM-selected incidents that are relevant to the context, events, problems, human factors, 
causes, or unspecified factors of the Cali accident. In this figure, an incident is counted as relevant if one or more of the 
analysts rated the incident as relevant. 
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Number of randomly selected incidents that are relevant to 
the context, events, problems, human factors, causes, 
or unspecified factors of the Cali accident 


Context 



Human factors 



Events 


Causes 



Problems Unspecified factors 



Figure 5. Number of randomly selected incidents that are relevant to the context, events, problems, human factors, 
causes, or unspecified factors of the Cali accident. In this figure, an incident is counted as relevant if one or more of the 
analysts rated the incident as relevant 
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Consensus among analysts: 

Number of incidents rated as relevant to the Cali accident 

by N of 3 analysts 


QUORUM-selected incidents Randomly selected incidents 



Figure 6. Consensus among analysts regarding the number of incidents relevant to the Cali accident, comparing results 
for QUORUM-selected incidents and those selected randomly . These pie charts show the number of QUORUM-selected 
incidents and randomly selected incidents that were rated as relevant by 3 of 3 analysts, 2 of 3 analysts , 1 of 3 analysts, 
and Oof 3 analysts . AH three analysts rated 48 of 100 QUORUM-selected incidents as relevant, compared with only 9 of 
the randomly selected incidents. At least two of three analysts rated 72 of 100 QUORUM-selected narratives as relevant , 
compared with only 16 of 100 randomly selected narratives. Of the 100 QUORUM-selected narratives, 84 were rated as 
relevant by at least one analyst Of the 100 randomly selected narratives, 41 were rated as relevant by at least one 
analyst. 


Degree of relevance: 

Number of incidents rated as relevant to the Cali accident 
in response to N of 6 assertions of relevance 


QUORUM-selected incidents Randomly selected incidents 



Figure 7 . Degree of relevance : Number of incidents rated as relevant tc the Cali accident in response to N of 6 assertions 
of relevance. Agreement with more than one assertion of relevance indicates a greater degree of relevance. These two 
pie charts show that the analysts found many of the QUORUM-selected incidents to be highly relevant, while few of the 
randomly selected incidents were highly relevant For example, in ratings of 45 of the QUORUM-selected incidents, each 
of the 6 statements of relevance was agreeable to one or more of the analysts, while only 3 of the randomly selected 
incidents were rated as relevant for ail six statements. Eighty-four of the 100 QUORUM-selected incidents were judged to 
have at least one relevant factor, while only 41 of the randomly selected incidents had at least one relevant factor. 
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Strength of analysts’ opinions 
that incidents are relevant to the Cali accident 

QUORUM-selected incidents Randomly selected incidents 


Analyst 1 


Analyst 1 



Analyst 2 


strongly disagree (1 3^^^^^^ s t r °ngly agree (12) 

disagree (2) 
somewhat 
disagree (9 )\\\] 

undecided (5) . Vv agree (3 1 ) 


somewhat agree (28) 



Analyst 2 


strongly disagree (51) 


disagree (7) — 



agree (1) 
somewhat agree (22) 

undecided (1 1 ) 

. $ 

somewhat disagree 

(8) 


Analyst 3 Analyst 3 



Figure 8. Strength of analysts' opinions that incidents are relevant to the Cali accident, comparing results for QUORUM- 
selected incidents and randomly selected incidents . Each pie chart shows the number of incidents whose highest rating 
across assertions of relevance was one of the following: 7: strongly agree, 6: agree, 5: somewhat agree, 4: undecided, 3: 
somewhat disagree , 2: disagree, or 1: strongly disagree . Overall, the analysts more strongly agreed that the QUORUM- 
selected incidents , rather than the randomly selected ones, are relevant to the Cali accident. Further, they more strongly 
disagreed that the randomly selected incidents are relevant 
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Appendix 1. 

Methods of modeling and ranking 




Appendix 1. This appendix describes a method of modeling bodies of text (including one or more documents , parts of 
documents, paragraphs , sentences, or word groups ) r generating query models, comparing models, and ranking 
collections of models according to their similarity to a query model, thereby obtaining the similarity-based ranking of 
documents in a collection . The degree of similarity is interpreted as the degree of relevance to the query. Steps marked 
with an asterisk (*) were not used in the Cali project 

Generate proximity- weighted co-occurrence model 

1. Identify the terms in a body of text. Each term can be one or more marks or characters, such as: a single 

punctuation mark; a sequence of marks and/or characters; a word; a linked set of marks, characters, or words; a 
tagged set of marks, characters, or words; or any combination of these. Ignore any terms that are on the stoplist, 
that is, terms that are not of interest in the analysis. If desired, special sections (e.g., titles, headers, tables, 
captions, tables of contents, indices, or other sections) may be differentiated from the body of text or 
incorporated into it. 

2. If desired, map specific classes of terms to other terms, as in mapping certain domain terms to non-word place 

holders. For example, "callback conversation," a phrase sometimes inserted into incident narratives by the 
ASRS, can be mapped to "non word nonword. M 

3. If desired, standardize the vocabulary, such as by converting multiple disparate forms of terms to a single 

representation, linking terms, tagging terms, changing case, or any combination of these. Once the vocabulary is 
standardized, return to step 1 . Terms especially appropriate for linking are those which should not, do not, or 
only rarely stand alone as individual terms. 

4. Map terms as needed to ease computer-based parsing and use of regular expressions. 

5. Generate a list of terms and their frequencies of occurrence in the body of text. 

6. Select a number of the most frequently occurring terms to serve as probe terms. If desired, exclude terms on the 

stoplist. 

7. If desired, use only the probe terms having particular tags or other distinguishing characteristics.* 

8. If desired, expand the list of probe terms by including synonyms of those already on the list, and scaling their 

frequencies accordingly.* 

9. At each occurrence of each probe term in the body of text, do steps 9. 1 through 9.4. 

9.1. Identify the terms in proximity to the probe term, to some distance from the probe term, as being among the terms- 

in-context of the probe term. The terms-in-context are considered to be in the context of each occurrence of the 
probe term. Instances of the probe term that are in the context of an occurrence of the probe term are considered 
to be terms-in-context of that occurrence. If desired, ignore terms-in-context that are on the stoplist. 

9.2. Consider each unique pair, consisting of a probe term and a term- in -context, as a relation having a value. 

9.3. For the current context, assign a value to each relation based on the proximity of each instance of the term-in- 

context to the probe term. This is a proximity value. 

9.4. For each unique relation found in this context, accumulate the sum of the proximity values. Each sum, accumulated 

across all contexts, is the relational metric value (RMV) of the relation within this particular body of text. 

10. List the unique relations, each with its relational metric value. 

11. Rank order the relations based on the magnitudes of the relational metric values. 

12. If desired, remove relations that are not of interest (e.g., "BUT NOT"). Remove relations containing stoplisted 

terms, if this was not done in step 9.1. 

13. Use all of the remaining relations, or select a number of the relations having the largest relational metric values. 

14. Use this list of relations to represent the body of text from which the relations were derived. 
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15. If desired, such a list of relations can be synthesized from scratch to represent the model of an idealized, 
hypothesized, or sought-after body of text.* 

Generate models of a collection of text 

1 . Identify a collection of bodies of text as a database of text. 

2. Derive a proximity- weighted co-occurrence model, as described in the previous section, of each body of text in the 

database of text to produce a database of models. Each body of text in the database of text corresponds to one 
model in the database of models. 

Generate query model 

1 . Select one or more bodies of text from any source, including but not limited to the database of text, to serve as the 

basis of a query. This is the query text. If desired, special sections (e.g., titles, headers, tables, captions, tables of 
contents, indices, or other sections) may be differentiated from the body of text or incorporated into it. 

2. If desired, map prominent query terms or term groups that are not found among the terms or term groups in the 

database of models to equivalent terms or term groups that are, in fact, found among the terms or term groups in 
the database of models. For example, map "first officer" to "FO" 

3. If necessary, match usage of uppercase and lowercase characters in the query text and the text to be queried. 

4. Derive a proximity-weighted co-occurrence model of the query text as described in the first section of this 

appendix to produce an initial-query model. 

5. For each relation in the initial-query model, find any and all instances of the corresponding relation in the database 

of models. For a relation A B in the initial-query model, where A is a probe term and B is a term-in-context, the 
relation A B or the relation B A in a database model constitutes a corresponding relation. Find the sum of the 
relational metric values of all instances of the corresponding relations in the database of models to derive the 
database RMV for that relation. 

6. Divide the RMV of each initial-query relation by the database RMV of that relation to derive the raw inverse query 

RMV. 

7. The probe term and term-in-context of the initial -query relation whose value is the raw inverse query RMV is a raw 

inverse query relation. 

8. The raw inverse query consists of a collection of raw inverse query relations. 

9. Multiply the RMV of each relation in the raw inverse query by the frequency of the relation's probe term in the 

query text and the frequency of the relation’s term-in-context in tne query text to produce a scaled RMV. 

10. If desired, scale all RMVs to convenient magnitudes. 

11. If desired, fine-tune the query model by removing relations that are not of interest.* 

12. Sort the query relations on the scaled RMV, and take some number of relations having the largest scaled RMVs to 

serve as the query model. 

13. The relation-by-relation product or sum of multiple query models can be used as a query model.* 

Compare query model to database models, and rank database models on similarity 

1 . Compare the query model to each model in the database of models. For each model in the database of models, do 
steps 1.1 through 1.3. 

1.1. For each query relation, if both terms are found in a database model relation, either as probe term or term-in- 

context, calculate the product of the RMV of the query relation and the RMV of the database model relation. 
This product is the intersection RMV, the resulting relation is the intersection relation, and the collection of 
intersection relations is the intersection model. (Functions might usefully modulate the terms and/or product.*) 

1 .2. Find the sum of the intersection RMVs to produce the raw similarity value associated with the database model and 

its corresponding body of text in the database of text. 
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1.3. If the database model has a non-zero raw similarity value, scale the raw similarity value to produce the similarity 
ranking value, using the scale factors in 1.3.1 through 1.3.4. 

1.3.1. One scale factor is calculated by finding the sum of the database model RMVs of relations that are also found in 
the query model, and dividing this value by the sum of all RMVs of the database model. This favors database 
models whose shared features are more central to the emphasis of the body of text from which the database 
model is derived. This tends to favor smaller bodies of text. 

1.3.2. A second scale factor is calculated by counting the number of terms in the database body of text that is 
represented by the current database model, and dividing this value by a number as large or larger than the 
number of terms in the largest body of text in the database. This favors larger bodies of text, counterbalancing 
tendencies of other scale factors to favor smaller bodies of text. (Clamping this factor might be useful.*) 

1 .3.3. An optional scale factor* is calculated by finding the sum of the query model RMVs of relations that are also 
found in the database model, and dividing this value by the sum of all RMVs of the query model. This favors 
database models whose shared features are more central to the emphasis of the text on which the query model is 
based. This tends to favor smaller bodies of text. 

1 .3.4. Another optional factor* is the number of relations that are shared by the query and the current database model. 
This favors bodies of text having a greater number of relations in common with the query text. 

2. The bodies of text having the largest similarity ranking values are most likely to be similar to the query text, and 

are most likely to be perceived as being relevant to the query, so the similarity ranking values may be 
interpreted as relevance ranking values. 

3. Sort identifiers of the bodies of text in the database according to the magnitude of their relevance ranking values, 

with larger values toward the head of the list. 

4. Bodies of text whose identifiers are nearer the head of the sorted list are more relevant to the query text. 

5. The most relevant bodies of text, as determined by their relevance ranking values and/or other interpretation 

method(s), may serve as the basis of subsequent query models and, by comparing one or more of the subsequent 
query models to the database models and ranking the database models on similarity, even more specifically 
relevant bodies of text may be obtained.* 

6. When bodies of text in the database of text are assigned multiple relevance ranking values (RRVs) with respect to 

multiple, separate query models, the product of the multiple RRVs can be used to relevance-rank the text on the 
logical "and” of the multiple queries.* 
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Appendix 2. 

Rating materials 




INSTRUCTIONS to RATERS 


Read each narrative. After reading each narrative, re-read each statement (A-F) below and respond to each statement by 
selecting one of the possible responses (1-7). On the response form, circle the response number you selected. 


Use a pencil with soft lead. Erase thoroughly if any changes must be made. 

After you read each narrative, it is very important that you re-read each statement (A-F) just before responding to it. So, 
after reading a narrative, you should re-read statement A, select one response from among the responses numbered 1-7, 
and circle that response on your response form. You should then re-read statement B, select among the responses, and 
circle that response on your response form. And so on. Be sure to respond to all six statements (A-F) for every narrative. 


Respond to each statement on its own merits. Ignore relationships among the statements. 
You may re-read all or part of a narrative as often as you wish. 

Please write your initials at the bottom of every page of the response form. 


Statements: 

A) In some ways, the context of this incident is similar to the context of the Cali accident. 

B) Some of the events of this incident are similar to some of the events of the Cali accident. 

C) Some of the problems of this incident are similar to some of the problems of the Cali accident. 

D) Some of the human factors of this incident are similar to some of the human factors of the Cali accident. 

E) Some of the causes of this incident are similar to some of the causes of the Cali accident. 

F) In some ways, this incident is relevant to the Cali accident. 


For each narrative, select one of these responses for each statement: 


1 

2 

3 

4 

5 

strongly 

disagree 

disagree 

somewhat 

disagree 

undecided 

somewhat 

agree 


6 

agree 


7 

strongly 

agree 


Appendix 2. Figure 1. Instructions to raters . 
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AND HIGH TERRAIN OF EITHER SIDE OF THE STAR ROUTING. WHEN I CHKED THE POINTS ON 
THE STAR AGAINST OUR CURRENT DIRECT THE VOR RTE OF FLT IT LOOKED LIKE WE WOULD 
BE VERY CLOSE TO THE STAR ROUTING. WHEN YOU ARE 200 MI OUT, A 15 MI DIFFERENCE 
IS BARELY NOTICEABLE. FURTHER CHKING OF THE AREA CHART AND OUR DIRECT THE VOR 
ROUTING SHOWED TERRAIN AT 14000 FT TO 1 1000 FT DIRECTLY ALONG OUR PATH. A 
SIMILAR ATC CLRNC HAPPENS VERY OFTEN FLYING INTO LIMA, PERU. MANY, MANY, MANY 
PLTS ARE NOT AWARE OF JUST HOW CRUCIAL IT IS NOT TO ACCEPT THESE DEADLY CLRNCS. 
PLEASE GET THE WORD OUT AGAIN. 

310989 

ACR X ISSUED CLRNC TO FLY HDG 230 DEGS, INTERCEPT THE SAN FRANCISCO 095 DEG 
RADIAL, DSND AND MAINTAIN 1 1000 FT.' I THEN REQUESTED THE FMS BRIDGE VISUAL RWY 
28R. TRACON CLRED US DIRECT ARCHI. ON ABOUT 9 PREVIOUS OCCASIONS, I HAD ARRIVED 
AT ARCHI WITHOUT ANY ADDITIONAL RTE CLRNC. IT IS NOT CLR THAT THE ORIGINAL 
CLRNC TO INTERCEPT THE SFO 095 DEG STILL APPLIES AFTER BEING AMENDED TO FLY 
DIRECT ARCHI. IN FACT, ON AT LEAST SOME OF THE PRIOR OCCASIONS, THE FMS WAS 
REQUESTED WITH TRACON WITH CHK-IN, AND THEIR ORIGINAL RTE CLRNC WAS SIMPLY TO 
FLY DIRECT ARCHI, LEAVING US TO ARRIVE OVER ARCHI WITH NO FURTHER RTE OR APCH 
CLRNC. OUR TURNS ON SOME OF THESE PREVIOUS OCCASIONS TO COURSE 275 WERE MADE ON 
ASSUMPTION! ON THIS OCCASION, APCHING ARCHI, I QUERIED 134.5 ABOUT OUR CLRNC 
AFTER ARCHI. BAY APCH RESPONDED, 'INTERCEPT THE FINAL APCH COURSE.' THIS WAS 
THE FIRST TIME I HAD HEARD THE TERM 'FINAL APCH COURSE' REFERRING TO THE SFO 
095 DEG RADIAL. I INQUIRED AGAIN FOR CLARIFICATION, THAT IS NOT CLR TO ME, 

SHOULD WE INTERCEPT THE SAN FRANCISCO 095 DEG RADIAL?' THE CTLR INSISTED ON 
USING ONLY THE TERMINOLOGY 'FINAL APCH COURSE’ AND WOULD NOT RESPOND OTHERWISE 
TO MY REPEATED ATTEMPTS TO RECONCILE MY UNCERTAINTY ABOUT INTERCEPTING THE SFO 
095 DEG OR INTERCEPTING, JUST BEYOND ARCHI, THE RWY 28R LOC OR CTRLINE. AT 
ARCHI, WE TURNED TO A COURSE OF 275 ON THE ASSUMPTION THAT IT WAS OUR 'FINAL 
APCH COURSE.' THIS AMBIGUITY WAS PROMPTLY RESOLVED ON THE SUBSEQUENT BAY APCH 
FREQ 135.65. THIS TERMINOLOGY DOESNT MEET THE LOGICAL DEFINITION OF A 'FINAL' 

APCH COURSE. THE FMS BRIDGE VISUAL RWY 28R APCH CHANGES COURSE 2 MORE TIMES 
BEFORE ARRIVING AT THE RWY. NOR DOES IT APPEAR TO MEE T THE AIM DEFINITION OF 
'FINAL APCH COURSE' AS PUBLISHED IN FLT OPS MANUAL. NOR DOES IT SERVE THE 
PURPOSE OF CLRLY DISTINGUISHING A CLRNC TO INTERCEPT THE VOR RADIAL OR THE RWY 
LOC AND CTRLINE. I SUGGEST 1) BAY APCH PROC BE CHANGED SO THAT A RTE CLRNC OR 
EXPECT FURTHER RTE CLRNC IS ISSUED WITH THE CLRNC TO FLY DIRECT ARCHI, AND 2) 


Appendix 2. Figure 2. Example page from the bound collection of 200 narrratives that were read by the analysts. 
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198046 


A: 

B: 

C: 

D: 

E: 

F: 


1 

1 

1 

1 

1 

1 


2 

2 

2 

2 

2 

2 


3 

3 

3 

3 

3 

3 


4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 


6 

6 

6 

6 

6 

6 


280530 
A: 1 2 

B: 1 
C: 1 
D: 1 
E: 1 
F: 1 


198841 
A: 1 2 

1 
1 
1 
1 
1 


B: 

C: 

D: 

E: 

F: 


3 

3 

3 

3 

3 

3 


4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 


197507 
A: 1 2 

B: 1 
C: 1 
D: 1 
E: 1 
F: 1 


2 

2 

2 

2 

2 


274027 


A: 

B: 

C: 

D: 

E: 

F: 


4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 


217430 
A: 1 2 

B: 1 
C: 1 
D: 1 
E: 1 
F: 1 


315261 
A: 1 2 

B: 1 

C: 1 
D: 1 

E: 1 
F: 1 


4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 


6 

6 

6 

6 

6 

6 


116871 
A: 1 2 

B: 1 
C: 1 
D: 1 
E: 1 
F: 1 


2 

2 

2 

2 

2 


280233 
A: 1 2 

1 
1 
1 
1 
1 


B: 

C: 

D: 

E: 

F: 


2 

2 

2 

2 

2 


3 

3 

3 

3 

3 

3 


4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 


301538 
A:1 2 

1 
1 
1 
1 
1 


B: 

C: 

D: 

E: 

F: 


2 

2 

2 

2 

2 


209811 


A: 

B: 

C: 

D: 

E: 

F: 


1 

1 

1 

1 

1 

1 


2 

2 

2 

2 

2 

2 


4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 


7 

7 

7 

7 

7 

7 


310143 
A: 1 2 

B: 1 

C: 1 
D: 1 
E: 1 
F: 1 


370656 
A: 1 2 

B: 1 
C: 1 
D: 1 
E: 1 
F: 1 


3 

3 

3 

3 

3 

3 


4 

4 

4 

4 

4 

4 


5 

5 

5 

5 

5 

5 


310989 
A: 1 2 


B 

C 

D 

E 

F 


1 

1 

1 

1 

1 


2 

2 

2 

2 

2 


Appendix 2. Figure 3. Example page from booklet of response forms. 


3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 


3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 


3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 


3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 


3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 


3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 


3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 
3 4 5 


6 

6 

6 

6 

6 

6 


6 

6 

6 

6 

6 

6 


6 

6 

6 

6 

6 

6 


6 

6 

6 

6 

6 

6 


6 

6 

6 

6 

6 

6 


6 

6 

6 

6 

6 

6 


6 

6 

6 

6 

6 

6 


7 

7 

7 

7 

7 

7 


7 

7 

7 

7 

7 

7 


7 

7 

7 

7 

7 

7 


7 

7 

7 

7 

7 

7 


7 

7 

7 

7 

7 

7 


7 

7 

7 

7 

7 

7 


7 

7 

7 

7 

7 

7 
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Appendix 3. 

Ratings of the 200 incident narratives read by the analysts, 
showing the QUORUM-calculated relevance ranking values (RRVs), 
QUORUM'S ratings derived from the RRVs, 
and ratings by each of the analysts 




Appendix 3. Table L Ratings of the 100 QUORUM-selected incident narratives. The incidents are sorted in order of the 
analysts' relevance ratings. The more highly rated incidents appear nearer the head of this list. The ASRS accession 
numbers of each incident are shown in column 1 (accnum). The relevance ranking values (RRVs) in column 2 were 
computed by QUORUM , based on the intersection (see example in table 3) between the query model (table 1) and each 
narrative model (see example in table 2). The QUORUM ratings in column 3 (labeled Q) are computed from the RRVs, 
as described in table 3. Ratings shown for each of the three analysts (Al, A2, A3) are their maximum ratings across the 
six assertions of relevance. This indicates whether the analyst saw any relevance in the narrative. For example, a rating 
of 7 indicates that the analyst " strongly agreed " with at least one of the six assertions of relevance. The last 1 6 incidents 
on this list, shown in italics, were rated by the analysts no higher than 4 ("undecided"), indicating that they found no 
relevance in these incidents. More detail is available elsewhere in this report about the top 1 1 incidents. The narrative of 
incident 310130 is shown in Fig. 2, and all of the analysts' ratings for that incident are shown in Table 4. The narratives 
and all of the analysts' ratings for the next 10 incidents, 368360-223467 , are shown in Appendix 4, Figures 1-10. 


accnum 

RRV 

0 Al 

A2 

A3 



310130 

3052 

6 

7 

7 

7 

160843 

31696 

368360 

11483 

6 

7 

7 

7 

217430 

4898 

272508 

6158 

6 

7 

6 

7 

219222 

3034 

280233 

7214 

6 

6 

7 

7 

279030 

2843 

315261 

4012 

6 

6 

7 

7 

297695 

5095 

347848 

9726 

6 

7 

6 

7 

302770 

4675 

349669 

2869 

6 

6 

7 

7 

305840 

4253 

363536 

10542 

6 

6 

7 

7 

359641 

9430 

310143 

24955 

7 

5 

7 

7 

82787 

2999 

310228 

4735 

6 

3 

7 

7 

363380 

3641 

223467 

3974 

6 

6 

6 

7 

153355 

2406 

334866 

4001 

6 

6 

6 

7 

174048 

4631 

351150 

8316 

6 

6 

7 

6 

201005 

2465 

184380 

6426 

6 

5 

7 

6 

238398 

3179 

212324 

6327 

6 

6 

5 

7 

251901 

2557 

274820 

3713 

6 

5 

6 

7 

307543 

7958 

306151 

2633 

6 

5 

6 

7 

317197 

3505 

335098 

2661 

6 

5 

7 

6 

330250 

2612 

226114 

4086 

6 

3 

7 

5 

325026 

7878 

142553 

5395 

6 

6 

6 

6 

362229 

2983 

146645 

7106 

6 

6 

6 

6 

156284 

4664 

224363 

2574 

6 

6 

6 

6 

310989 

14024 

310373 

17066 

6 

6 

6 

6 

156414 

4762 

334006 

4234 

6 

6 

6 

6 

184446 

7494 

352618 

4305 

6 

6 

6 

6 

198046 

3758 

116871 

7215 

6 

6 

6 

5 

242545 

2982 

117306 

6867 

6 

6 

6 

5 

275413 

4687 

244767 

2684 

6 

5 

6 

6 

335430 

5837 

296506 

2735 

6 

6 

6 

5 

346137 

5557 

301760 

8371 

6 

6 

6 

5 

361956 

5096 

321136 

3124 

6 

6 

5 

6 

365456 

3330 

341815 

4676 

6 

6 

6 

5 

335282 

9002 

353338 

7910 

6 

2 

6 

6 

300252 

7941 

140711 

3067 

6 

6 

1 

6 

329185 

3011 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: 


7 

5 

6 

5 

342838 

4584 

6 

5 

5 

3 

6 

5 

6 

5 

358123 

6317 

6 

3 

5 

5 

6 

5 

6 

5 

84811 

5349 

6 

5 

5 

3 

6 

5 

5 

6 

115883 

2736 

6 

5 

5 

2 

6 

5 

5 

6 

354277 

6222 

6 

2 

5 

5 

6 

5 

6 

5 

307161 

10147 

6 

5 

5 

1 

6 

6 

5 

5 

303310 

2323 

6 

5 

4 

3 

6 

6 

5 

5 

264952 

6695 

6 

5 

3 

3 

6 

5 

6 

5 

279493 

6803 

6 

5 

3 

3 

6 

5 

4 

6 

350190 

3118 

6 

5 

3 

3 

6 

6 

5 

3 

355364 

2948 

6 

3 

5 

3 

6 

3 

6 

5 

242560 

2612 

6 

2 

5 

3 

6 

3 

5 

6 

249654 

4542 

6 

2 

3 

5 

6 

3 

6 

5 

309352 

3498 

6 

2 

3 

5 

6 

3 

5 

6 

282707 

3565 

6 

5 

1 

3 

6 

3 

6 

5 

99108 

3018 

6 

3 

1 

5 

6 

5 

6 

3 

260432 

9963 

6 

4 

3 

1 

6 

3 

6 

5 

280922 

9001 

6 

2 

4 

2 

6 

5 

6 

2 

325365 

4521 

6 

3 

3 

3 

6 

2 

6 

5 

308422 

4712 

6 

2 

3 

3 

6 

2 

4 

6 

228422 

3901 

6 

3 

1 

3 

6 

4 

2 

6 

355]88 

2659 

6 

3 

1 

3 

6 

5 

5 

5 

313511 

5440 

6 

2 

2 

3 

6 

5 

5 

5 

112422 

2386 

6 

1 

3 

2 

6 

5 

5 

5 

281636 

3187 

6 

3 

1 

2 

6 

5 

5 

5 

370656 

6793 

6 

3 

1 

2 

6 

5 

5 

5 

332870 

2503 

6 

3 

1 

1 

6 

5 

5 

5 

148439 

2708 

6 

2 

1 

2 

6 

5 

5 

5 

342160 

3508 

6 

2 

1 

2 

6 

5 

5 

5 

325432 

10172 

6 

2 

1 

1 

6 

5 

5 

5 

340978 

2313 

6 

2 

1 

1 

6 

5 

4 

5 

360500 

3416 

6 

2 

1 

1 

6 

3 

5 

5 







6 

3 

5 

5 








undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 
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Appendix 3. Table 2. Ratings of the 100 randomly selected incident narratives. The incidents are sorted in order of the 
analysts’ relevance ratings. The more highly rated incidents appear nearer the head of this list. The ASRS accession 
numbers of each incident are shown in column 1 (accnum). The relevance ranking values (RRVs) in column 2 were 
computed by QUORUM \ based on the intersection (see example in table 3/ between the query model (table 1) and each 
narrative model (see example in table 2). The QUORUM ratings in column 3 (labeled Q) are computed from the RRVs, 
as described in table 3. Ratings shown for each of the three analysts (Al f A2, A3) are their maximum ratings across the 
six assertions of relevance. This indicates whether the analyst saw any relevance in the narrative. For example , a rating 
of 7 indicates that the analyst "strongly agreed" with at least one of the six assertions of relevance. 


accnum RRV O A1 A2 A3 


137942 

102 

4 

5 

5 

7 

329210 

0 

1 

1 

5 

'S 

280530 

0 

1 

3 

1 

1 

168420 

612 

5 

6 

5 

5 

352880 

0 

1 

1 

5 

2 

288665 

0 

1 

1 

1 

3 

184917 

0 

1 

5 

5 

6 

374411 

0 

1 

5 

1 

2 

299682 

0 

1 

3 

1 

1 

232995 

0 

1 

5 

6 

2 

98676 

0 

1 

5 

1 

'2 

339510 

0 

1 

3 

1 

1 

276144 

0 

1 

2 

5 

6 

217405 

19 

4 

1 

1 

5 

349077 

7 

3 

3 

1 

1 

185995 

4 

3 

2 

1 

6 

289604 

0 

1 

5 

1 

1 

231377 

0 

1 

1 

2 

2 

369400 

0 

1 

1 

1 

6 

326579 

14 

3 

1 

5 

1 

273126 

0 

1 

2 

2 

1 

121942 

0 

1 

5 

5 

5 

334370 

1 

2 

3 

4 

3 

311780 

0 

1 

2 

1 

2 

123523 

6 

3 

5 

5 

5 

269069 

0 

1 

2 

4 

3 

336170 

0 

1 

2 

1 

2 

133697 

0 

1 

5 

5 

5 

224527 

0 

1 

2 

4 


363445 

0 

1 

2 

1 

2 

137377 

2 

3 

5 

5 

5 

135427 

0 

1 

1 

4 

1 

197399 

0 

1 

1 

1 

2 

299590 

32 

4 

5 

5 

5 

167263 

0 

1 

2 

3 

3 

199234 

0 

1 

2 

1 

1 

317302 

2 

3 

5 

5 

5 

205316 

11 

3 

3 

3 

2 

221398 

20 

4 

2 

1 

1 

120627 

6 

3 

3 

5 

5 

348150 

0 

1 

3 

2 

3 

227582 

0 

1 

2 

1 

1 

124286 

0 

1 

5 

5 

3 

164488 

0 

1 

3 

1 

3 

233097 

0 

1 

2 

1 

1 

294893 

0 

1 

5 

5 

2 

221067 

57 

4 

3 

3 

1 

236441 

0 

1 

2 

1 

1 

301538 

0 

1 

5 

2 

5 

236993 

1 

2 

3 

1 

3 

243432 

7 

3 

2 

1 

1 

80231 

0 

1 

5 

5 

2 

1 19934 

0 

1 

2 

3 

o 

250185 

0 

1 

2 

1 

1 

163375 

0 

1 

3 

4 

5 

88123 

10 

3 

3 

2 

o 

266321 

0 

1 

2 

1 

1 

226033 

0 

1 

3 

4 

5 

129682 

0 

1 

2 

1 

3 

285669 

0 

1 

2 

1 

1 

92389 

0 

1 

5 

4 

3 

145934 

0 

1 

2 

3 

1 

301427 

0 

1 

2 

1 

1 

197507 

48 

4 

5 

4 

2 

164303 

0 

1 

2 

1 

■> 

310662 

0 

1 

2 

1 

1 

296006 

0 

1 

5 

4 

2 

176239 

0 

1 

3 

1 

2 

319332 

3 

3 

2 

1 

1 

93994 

0 

1 

5 

4 

2 

305310 

0 

1 

1 

3 

2 

345249 

0 

1 

2 

1 

1 

294068 

0 

1 

1 

4 

5 

307714 

0 

1 

2 

1 

3 

358446 

0 

1 

2 

1 

1 

125733 

0 

1 

3 

2 

5 

107473 

0 

1 

3 

1 

L 

363100 

16 

3 

2 

1 

1 

184142 

0 

1 

5 

3 

2 

137871 

2 

3 

1 

1 

3 

150685 

0 

1 

1 

1 

1 

357280 

60 

4 

2 

5 

3 

154070 

0 

1 

3 

1 

l 

159808 

0 

1 

1 

1 

1 

1 19343 

76 

4 

1 

5 

3 

181017 

0 

1 

3 

1 

t 

173602 

0 

1 

1 

1 

1 

236719 

0 

1 

3 

5 

1 

196326 

0 

1 

3 

1 

l 

245613 

0 

1 

1 

1 

1 

274027 

58 

4 

5 

3 

1 

198841 

0 

1 

3 

1 

l 

272741 

0 

1 

1 

1 

1 

297200 

12 

3 

3 

5 

1 

209811 

0 

1 

3 

1 

l 

282765 

0 

1 

1 

1 

1 

324025 

0 

1 

3 

5 

1 

211265 

0 

1 

3 

1 

i 







114244 

1 

2 

5 

2 

2 

249845 

0 

1 

3 

1 

1 . 








(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 
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Appendix 4. 

Ten of the incidents rated as 
highly relevant to the Cali accident 




368360 

AFTER DEPARTING FRG ARPT, ENRTE TO OUR FIRST DEP FIX, WE WERE INSTRUCTED BY ATC 
TO FLY A HDG OF 360 DEGS BECAUSE OF A LINE OF TSTMS JUST W OF OUR DEP FIX. THE 
CTLR TOLD US TO MAINTAIN FL280 AND CONTACT ZNY FOR A RERTE. WE CONTACTED ZNY, 
WHO ISSUED A NEW ROUTING TO US. AS I WAS ENTERING THE DATA IN THE FMS IT BECAME 
CLR TO ME THAT THIS ROUTING TO PWK (OUR DEST) WAS INVALID, AS THE FMS WAS NOT 
TAKING THE INFO. I ASKED MY FO TO CLARIFY THE ROUTING WITH ATC. HE ATTEMPTED TO 
DO SO BUT WE WERE GIVEN A FREQ CHANGE AT THIS TIME TO ZBW. ON OUR INITIAL CALL 
TO ZBW, HE TOLD US TO FLY DIRECT TO ALB VOR, CLB TO FL310 AND WHEN ABLE PROCEED 
DIRECT SYR VOR. AT THIS TIME I AGAIN ASKED MY FO TO CLARIFY THE ROUTING AFTER 
SYR. WE WERE AGAIN GIVEN INVALID ROUTING (RTE BREAK ON FMS). AT THIS POINT MY 
FO BECAME ENGROSSED IN LOOKING ON OUR HIGH AND LOW ALT ENRTE CHARTS TO FIND 
WHERE THE PROB WAS. I TRIED AGAIN TO ENTER ROUTING ON THE FMS (THIS TOOK APPROX 
2 MINS). I LOOKED UP TO SEE MY FLT INSTS AND AT THIS TIME NOTED THE ALTIMETER 
READING FL3 12 AND CLBENG. I IMMEDIATELY DISCONNECTED THE AUTOPLT AND ATTEMPTED 
TO DSND TO FL310, BUT OUR 1000 FPM RATE OF CLB CARRIED US TO FL313 BEFORE 
CORRECTIVE ACTION WAS INITIATED. APPROX 5 SECONDS LATER ZBW TOLD US TO MAINTAIN 
FL310. 1 LOOKED AT MY TCASII DISPLAY WHICH WAS IN THE 'LOOP UP' MODE BUT SAW NO 
CONFLICTING TFC WITHIN A 40 MI RADIUS. ATC FINALLY GAVE US A RTE THAT WAS VALID 
AND WE CONTINUED ON TO PWK WITHOUT FURTHER INCIDENT. I BELIEVE THAT THE 
COMPLEXITY OF FMS PROGRAMMING IS NOT ADDRESSED IN INITIAL TRAINING AT SCHOOL 
BECAUSE EACH ACFT HAS DIFFERENT EQUIP. HOWEVER, THIS LEAVES THE FLC TO 'LEARN 
AS THEY FLY.' THIS EFFECTIVELY TOOK MY FO OUT OF THE LOOP IN THAT IF HE WAS 
PROGRAMMING THE FMS, I COULD HAVE CONCENTRATED MORE ON MONITORING THE ACFT. I 
SHOULD HAVE LET THE FO FLY THE ACFT WITH THE AUTOPLT RATHER THAN ME DO ALL THE 
TASKS. THE ENTIRE CREW WAS DISTR, AND WE BOTH FAILED TO MONITOR THE PERFORMANCE 
OF THE ACFT. I SHOULD HAVE JUST PUT MY HSI IN THE VOR MODE RATHER THAN DISPLAY 
FMS COURSE INFO. THIS WOULD HAVE ALLOWED US TO FOCUS MORE ON THE ACFT. I 
BELIEVE, HOWEVER, THAT ALL GA PLTS WITH FMS'S ON BOARD ATTEND FMS SPECIFIC 
SIMULATOR TRAINING ON THEIR UNITS BEFORE OPERATING AN ACFT WITH THE UNIT 
INSTALLED. THERE IS NO REG THAT GOVERNS THIS OTHER THAN FAR PART 91 WHICH 
STATES I MUST BE FAMILIAR WITH THE OP OF ALL EQUIP. SO I AM LEFT TO READ A BOOK 
ON THE FMS AND THEN GO FLY IT IN REAL LIFE. THIS IS UNACCEPTABLE. 

Analysts' ratings of each assertion of relevance 


report 

number analyst 
368360 analyst 1 
analyst 2 
analyst 3 


A: similar 
context 

B: similar 
events 

C: similar 
problems 

D: similar 
human factors 

E: similar 
causes 

F: unspecified 
relevance 

7 

6 

6 

6 

6 

6 

2 

5 

7 

6 

6 

5 

6 

6 

7 

6 

6 

5 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 1. Narrative of ASRS incident report number 368360, one of the narratives rated as highly relevant to 
the Cali accident Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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272508 

I MISSED THE XING RESTRICTION OF 10000 FT AT ’BUMBY' ON THE BATSN.BATSN3 ARR TO 
HOU. WE HAD BEEN MANEUVERING AROUND TSTMS AND HAD RECEIVED A FEW ROUTING 
CHANGES, THE LATEST OF WHICH WAS DIRECT TO DAS THEN TO THE ARPT.’ AS WE APCHED 
HOU, WE WERE CLRED DIRECT TO BUMBY AND THE REMAINDER OF THE ARR. I STARTED THE 
AIRPLANE DIRECT TO BUMBY WITH THE FMS AND THEN INSERTED THE XING RESTRICTION OF 
10000 FT ON THE LEGS PAGE. I THEN SELECTED THE BATSN-3 ARR FOR USE BY THE FMS. 

WHAT I FAILED TO NOTICE WAS THAT BY INSERTING THE ARR IN THE FMS, THE COMPUTER 
DUMPED THE XING RESTRICTION I HAD INSERTED JUST A FEW MOMENTS EARLIER. AT THIS 
POINT, I FELT COMFORTABLE WITH THE POS OF TOP OF DSCNT' POINT AND PROCEEDED TO 
LOAD IN THE APCH TO RWY 4 (ILS), WHICH HAD CHANGED FROM RWY 12R WITH A NEW ATIS 
MESSAGE. AT 15 MI PRIOR TO THE DSCNT POINT, I STARTED DOWN AND INTERCEPTED THE 
DSCNT PROFILE WELL AHEAD OF SCHEDULE. DURING THE DSCNT I BRIEFED THE APCH AND 
SET UP THE NAV FOR THE ILS. THROUGH ABOUT 17500 FT, APCH CTL ASKED IF WE WOULD 
MAKE THE BUMBY RESTRICTION (10000 FT) AND IT WAS IMMEDIATELY OBVIOUS THAT WE 
WOULD NOT AS THE DSCNT LINE WE WERE ON NO LONG WAS USING THE BUMBY RESTRICTION 
FOR COMPUTATION. THE CTLR STATED THAT IT WAS NOT A PROB AND THAT WE SHOULD JUST 
KEEP OUR SPD UP AND PROCEED ON THE ARR. THE LNDG WAS COMPLETED WITHOUT FURTHER 
INCIDENT OR DIFFICULTY. THE CAUSE, I BELIEVE, WAS A COMBINATION OF COCKPIT 
MGMNT OVERLOAD DURING THE APCH PHASE COUPLED WITH AN OVERCONFIDENCE IN THE FMS 
TO PRESENT VALID DSCNT PROFILE INFO. I ALLOWED MYSELF TO GET TOO BUSY DURING 
THE DSCNT TO MAKE ESSENTIAL XCHKS TO CONFIRM THE FMS WAS WORKING AS ADVERTISED. 
THE CORRECTION. ALWAYS DOUBLE CHK THE FMS DATA AGAINST OTHER AVAILAABLE NAV 
DATA TO INSURE THAT YOUR PROGRAMMING IS CORRECT AND THAT THE ACFT IS FOLLOWING 
ACCURATE FMS GUIDANCE. OVERCONFIDENCE IN THE FMS AND INCREASED WORKLOAD IN THE 
COCKPIT DURING BAD WX AND APCH PREPARATION IS NO EXCUSE FOR SOUND PILOTAGE AND 
THE MAINT OF SITUATIONAL AWARENESS. 


Analysts' ratines of each assertion of relevance 


report 

number analyst 
272508 analyst 1 
analyst 2 
analyst 3 


A: similar 
context 

B: similar 
events 

C: similar 
problems 
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human factors 
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causes 

F: unspecified 
relevance 
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7 
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6 

5 

6 
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7 

7 

6 
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(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 2. Narrative of ASRS incident report number 272508, or e of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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280233 

APCH CTL CLRED US TO 7200 FT INITIAL APCH ALT FOR RWY 16R AT RNO. BECAUSE OUR 
PRESENT ALT WAS HIGHER THAN NORMAL, I CALLED FOR ’GEAR-DOWN' AND EXTENDED SPD 
BRAKES TO AID DSCNT. I INTERCEPTED RWY 16R LOC AND TRACKED INBOUND WHILE STILL 
DSNDING. SINK RATE ABOUT 3000 FPM. AT 8200 FT THE ALT ALERT SIGNALED AND THE 
COPLT AND I EXCHANGED ACKNOWLEDGEMENTS. I RAISED THE NOSE TO ARREST SINK RATE 
AND DECELERATE AT 7200 FT. APCHING ZERO-FLAP VMA. I APPLIED THRUST AND NOTICED 
I WAS USING CONSIDERABLY MORE THAN NORMAL TO MAINTAIN LEVEL FLT. AIRSPD WAS 210 
KT AND DECAYING. LNDG WT WAS APPROX 120000 LBS AND VMA ABOUT 180 KTS. AT APPROX 
200 KTS, I GOT THE STALL SHAKER WITHOUT ANY UNUSUALLY HIGH NOSE ATTITUDE. I 
INSTINCTIVELY LOWERED THE NOSE AND ADDED THRUST. WE WERE NOW DSNDING BELOW 7200 
FT AND BELOW GS PRIOR TO THE OM. AT 6500 FT THE GPWS ISSUED A TERRAIN' 

WARNING. THE FE THEN ALARMED ME THE SPD BRAKES WERE STILL EXTENDED. I 
IMMEDIATELY RETRACTED THEM, CALLED FOR 'FLAPS-2 DEGS' AND CALLED THE ARPT IN- 
SIGHT.' WE WERE THEN CLRED FOR A VISUAL TO RWY 16L AND MADE A NORMAL APCH AND 
LNDG. ANALYZING THESE EVENTS, I HAD FORGOTTEN THE SPD BRAKES WERE DEPLOYED. THE 
LNDG GEAR WAS DOWN TO AID DSCNT BUT WAS NOW A LARGE DRAG DEVICE IN LEVEL FLT. 
ADDITIONALLY, I WAS IN A NON-STANDARD APCH PROFILE. GETTING THE STALL SHAKER AT 
200 KTS CONFUSED ME AND MY INSTINCT TO LOWER THE NOSE AND ADD THRUST WAS 
TRIGGERED ALTHOUGH I WAS STILL LOSING ALT OVER MOUNTAINOUS TERRAIN. THESE 
ACTIONS PREVENTED A STALL BUT ACCELERATED ALT LOSS. THE RNO AREA IS MOUNTAINOUS 
AND WE GOT WITHIN GPWS TERRAIN WARNING RANGE AT 6500 FT. FORTUNATELY, MY FE 
CALLED MY ATTN TO THE SPD BRAKES. I HAVE BEEN THOROUGHLY TRAINED IN TERRAIN 
AVOIDANCE MANEUVERS BUT DID NOT EXECUTE THEM SINCE I WAS FOCUSED ON AIRSPD AND 
ALT CTL. OUR SCHEDULE WAS MSP TO DFW, DFW TO OAK, OAK TO RNO. WE ALL RECEIVED 
15 HRS REST PRIOR TO THESE TRIPS YET ALL OF US FELT EXTREMELY LETHARGIC AND 
FATIGUED UPON ARR AT OAK. WORKING 1 1 :28 HRS OF SCHEDULED DUTY AT THIS TIME OF 
DAY, I BELIEVE, IS THE MAJOR CAUSE OF THIS FATIGUE FEELING, RESULTING IN A 
GROSS IMPAIRMENT OF OUR JUDGEMENT. ADEQUATE CREW REST PRIOR TO THIS TRIP 
PAIRING IS NOT AN ANSWER. A SHORTER DUTY PERIOD IS REQUIRED. 


Analysts' ratings of each assertion of relevance 


report 

number analyst 
280233 analyst 1 
analyst 2 
analyst 3 


A: similar 
context 
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(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 3. Narrative of ASRS incident report number 280233, one of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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315261 

DURING OUR INITIAL APCH INTO ONT WE RECEIVED THE FOLLOWING CLRNC, PRO CEED 
DIRECT TO THE PETIS NDB, MAINTAIN 4200 FT TO PETIS, CLREE' ILS RWY 26L APCH. 

WHILE MY FO WAS READING BACK THE CLRNC I ENTERED DIRECT 'FF26L' INTO THE FMC 
AND PROCEEDED TO FLY INBOUND. A FEW MINS LATER APCH CTL CALLED AND SAID WE WERE 
4 NM L OF PETIS AND TO TURN R TO A 320 DEG HDG, MAINTAIN 4200 FT. AT THIS POINT 
I REALIZED THAT I HAD BEEN FLYING DIRECT TO THE OM (FONTA) INSTEAD OF PETIS 
NDB. WE THEN RECEIVED ANOTHER APCH CLRNC AND CONTINUED UNEVENTFULLY TO THE 
ARPT. WE WERE IN VFR CONDITIONS THE ENTIRE TIME AND NEVER HAD ANY TFC OR 
TERRAIN CONFLICTS. THE FMS NAV DATA BASE LISTS THE OM (FONTA) AS FF26L INSTEAD 
OF JUST FONTA. MOST NDB’S ARE COLLOCATED WITH THE OM, REFERRED TO AS LOM’S. 

HUMAN FACTORS BEING WHAT THEY ARE, I SAW FF26L AND USED THAT WAYPOINT THINKING 
AT THAT MOMENT THEY WERE COLLOCATED AND I WAS FLYING TO PETIS. ACTUALLY THEY 
ARE ABOUT 5 NM APART. IF THE FMS HAD SHOWN FONTA AS THE OM WAYPOINT INSTEAD OF 
FF26L I THINK IT WOULD HAVE BEEN CLR THAT THEY ARE IN F ACT 2 DIFFERENT POINTS. 

THERE DOESNT SEEM TO BE ANY CONSISTENCY WITH THE FMS APCH DATA BASE. SOME 
APCHS SHOW THE OM NAME AND OTHERS USE THE FF (FINAL FIX) FORMAT. THEY SHOULD 
ALL USE THE CORRECT FIX NAMES. ADDING TO THE CONFUSION THE APCH DATABASE SHOWS 
PETIS AS SBNB. OUR SOP DICTATES THE PNF MAKE ALL EXECUTABLE ENTRIES TO THE FMS. 
THIS KEEPS BOTH PLTS ’IN THE LOOP.’ I WAS HAND FLYING AND SINCE WE WERE VERY 
BUSY AT THE TIME, I MADE THE ENTRY TO SAVE TIME. FOLLOWING SOP MAY HAVE 
PREVENTED THE MIX-UP. FOLLOW SOP AND STAY ALERT, AND PLEASE MAKE THE DATA BASES 
MORE USER FRIENDLY! 


Analysts' ratings of each assertion of relevance 


report 

number 

315261 


A: similar B: similar 


analyst context events 

analyst 1 6 6 

analyst 2 6 6 

analyst 3 6 5 


C: similar 

D: similar 

E: similar 

problems 

human factors 

causes 

6 

5 

5 

7 

6 

6 

6 

6 

7 


F: unspecified 

relevance 

6 

6 

6 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 4. Narrative of ASRS incident report number 315261, one of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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347848 

3 MINS PRIOR TO TOP OF DSCNT, CAPT HAD TO LEAVE FLT DECK TO ATTEND TO 
PHYSIOLOGIC NEEDS. FO WAS PF FOR LEG AND HAD CIVET ARR PROGRAMMED IN FMS PRIOR 
TO CAPT LEAVING. HE HAD NOT SELECTED THE RWY AT THAT TIME. WHEN CAPT RETURNED, 
ACFT WAS PASSING FL280 TO MAKE 140A180B AT CIVET, LNAV/VNAV WAS IN USE. WE WERE 
CTRED ON LOC COURSE ON 109.9, BUT FO HAD NOT SELECTED EITHER HSI TO 'ILS' MODE 
SO AS TO RECEIVE DME AND NEITHER VOR WAS IN MANUAL -- TOTAL RELIANCE ON THE 
'MAGIC.' CAPT BRIEFLY SELECTED ILS AND VERIFIED DME/LOC CTRED AND WAS OCCUPIED 
IN COMPLETING APCH SETUP IN FMS, GETTING ATIS, PLANNING DSCNT CHK AND BRIEFING 
THE APCH FOR FO. FO WAS BEHIND AND NOT CATCHING UP. SEQUENCE IN FMS PROGRAMMING 
WAS SELECTING RWY 25L, LNAV DIRECT DOWNE, THEN BACK TO DEP/ARR PAGE TO RESELECT 
RWY 25L, CIVET, ARNES TRANSITION. THIS DROPPED DOWNE AS ACTIVE WAYPOINT, AND 
STARTED JET IN TURN BACK TO ARNES. FO DID NOT HAVE LOC SELECTED!? LAX APCH GAVE 
A TURN TO 220 DEG TO REJOIN RWY 25L LOC. THATS WHEN CAPT NOTED FO’S STATE OF 
'BEHINDNESS,' AND TALKED HIM THROUGH THE STEPS NECESSARY TO COMPLETE APCH 'RAW 

data; after that, there was barely enough time to complete the chklists, brief 

THE APCH, AND GET THE JET ON THE GND. FO HAS ERRONEOUSLY SELECTED APCH PRIOR TO 
10 DME, PER NOT AMS WHEN INSTRUCTION HAD BEEN SELECT LOC AND GOT LOW (AS A 
CONSEQUENCE OF THAT) BY 200 FT AT HUNDA ( 1 DOT LOW). CAPT WAS TASK SATURATED AS 
WELL, TRYING TO SUPPORT FO AND NOT TAKE THE ACFT OR TAKE US OUT OF SEQUENCE 
SINCE MANY PAX HAD TIGHT INTL CONNECTIONS (ALREADY 30 MINS LATE). THAT WAS THE 
CAPES MISTAKE, ALTHOUGH WE PULLED IT OUT THIS TIME. BUT BOY, IT WAS NOT 
PRETTY!! CAPT AND FO WERE BOTH UNDER 100 HRS AND NEW TO JET - PLAYED SOME PART 
IN EVENTS DESCRIBED. 


Analysts' ratings of each assertion of relevance 


report 

number 

analvst 

A: similar 
context 

B: similar 
events 

C: similar 
problems 
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human factors 
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causes 
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347848 

analyst 1 
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analyst 2 
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analyst 3 
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6 
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(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 5. Narrative of ASRS incident report number 347848, one of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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349669 

I UNDERSTAND THAT YOUR JURISDICTION MORE THAN LIKELY DOES NOT INCLUDE 
NICARAGUA, HOWEVER, I BELIEVE THIS WAS A CLASSIC SIT OF A FOREIGN ATC LANGUAGE 
BARRIER. THE WX AROUND MANAGUA WAS APPROX 1500 FT OVCST AND 10 MI, TSTMS 
SURROUNDING THE FIELD AND AT THE FIELD ITSELF. THE RIDE ON THE DSCNT WAS 
NOTHING MORE THAN LIGHT TURB WITH VERY OCCASIONAL MODERATE CHOP AND MODERATE 
RAIN. I HAD PROGRAMMED THE FMC TO PLAN ON A DSCNT TO ARRIVE 35 NM FROM MANAGUA 
AT 10000 FT MSL AND 250 KTS. WE WERE ON FLT PLAN AIRWAY A502 TO MANAGUA. AS THE 
FO AND MYSELF LOOKED AT THE APCH PLATE WE NOTICED THAT THERE WERE 1 OF 2 WAYS 
TO EXECUTE THE VOR RWY 9 APCH. HE HAD MENTIONED TO ME THAT HE HAD ALWAYS USED 
THE PROC OF INTERCEPTING THE PUBLISHED 10 DME ARC AND UTILIZING THAT METHOD TO 
RWY 9. AS I HAD NEVER BEEN TO MANAGUA BEFORE AND NOTING THE PRESENCE OF AIRWAY 
A502 DEPICTED ON THE CHART AS THE START POINT OF THE 10 DME ARC I AGREED THAT 
THIS WAS PROBABLY WHAT WE WOULD GET. LET ME NOTE THAT MANAGUA IS A NON RADAR 
APCH CTL ENVIRONMENT. THE CTLR PROCEEDED TO STEP US DOWN FROM 1 1000 FT MSL TO 
5000 FT MSL. PRIOR TO THE TURN OFF OF THE AIRWAY TO INITIATE THE 10 DME ARC SHE 
SAID RPT 5 DME RWY 9.' THERE WAS NO STATEMENT FROM THE CTLR TO 'EXPECT APCH 
CLRNC AT THE VOR,' CROSS THE VOR AT 5000 FT, RPT 5 DME,' 'CROSS THE VOR AT 
5000 FT, CLRED FOR THE APCH.' ONLY 'RPT 5 DME RWY 9.' WE BOTH ASSUMED WRONGLY 
THAT SHE HAD MEANT RPT 5 DME OUT ON FINAL ON THE APCH TO RWY 9. SO WE COMMENCED 
FLYING THE ARC, TRANSITIONING FROM THE AIRWAY AND COMPLYING WITH THE CHARTED 
STEPDOWN FIXES ASSOCIATED WITH THE VOR 9 APCH. AS WE WERE TURNING FINAL AT 
APPROX 9 DME FROM THE RWY THE CTLR ASKED OUR POS AND ALT. WE RESPONDED WITH, 

’ON FINAL, 9 DME AT 2700 FT AS PUBLISHED.' SHE THEN SAID THAT WE WERE TO HAVE 
CALLED 5 DME FROM THE VOR AND CROSSED THE VOR AT 5000 FT. THERE WAS NO OTHER 
TFC IN THE AREA. WE LANDED WITHOUT ANY INCIDENT AND THE CTLR NEVER QUESTIONED 
FURTHER. WE WERE HANDED OFF TO TWR AND GND CTL AND NOTHING MORE WAS MENTIONED 
AT ALL. IN FURTHER CONVERSATION BTWN THE FO AND MYSELF WE BOTH CAME TO THE SAME 
CONCLUSION THAT ALL THAT WOULD HAVE HAD TO HAVE BEEN SAID WAS 'CROSS THE VOR AT 
5000 FT, RPT 5 DME AND CLRED FOR THE APCH,' OR CLRED DIRECT TO THE MANAGUA 
VOR, MAINTAIN 5000 FT, AND RPT 5 DME.' WE ARE NOT PLACING BLAME ON ANYONE IN 
PARTICULAR AS I BELIEVE ALL OF US WERE TO FAULT TO A POINT. JUST SIMPLE MISCOM. 


Analysts' ratings of each assertion of relevance 


report 

number analyst 
349669 analyst 1 
analyst 2 
analyst 3 
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(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 6. Narrative of ASRS incident report number 349669, one of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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363536 

WE WERE ON A VECTOR FOR FINAL APCH COURSE TO ILS 3 AT CPR. DUE TO OUR HIGH ALT 
THE CTLR VECTORED US ACROSS FINAL ON A NE HDG. ACFT ANTI-ICE SYS WERE OPERATIVE 
AND SPD BRAKES WERE DEPLOYED TO KEEP AIRSPD BELOW 250 KIAS IN DSCNT. WE WERE 
CLRED TO DSND TO 7400 FT MSL. CTLR GAVE US A VECTOR TOWARDS THE S, R TURN IN 
ORDER TO MANEUVER US AROUND TO REINTERCEPT FINAL. SHORTLY AFTER THIS THE GPWS 
ALERTED US TO TERRAIN, TERRAIN.' AN IMMEDIATE CLB WAS INITIATED AND THE ACFT 
WAS SUBSEQUENTLY LANDED SAFELY. CONTRIBUTING FACTORS: 1) CTLR VECTORED US AT AN 
ALT BELOW MVA. 2) POOR SITUATIONAL AWARENESS BY FLC. 3) CREW DISTR BY 
CONVECTIVE ACTIVITY IN AREA AND ICING. 4) TOO MUCH TRUST PUT IN CTLR BY FLC. 
CALLBACK CONVERSATION WITH RPTR REVEALED THE FOLLOWING INFO: RPTR CALLED BACK 
WITH THE FOLLOWING: THE FLC BRIEFING WAS DONE BUT IT WAS ACCOMPLISHED ALONG 
WITH THE DSCNT CHKLIST DURING THE APCH AND VECTOR FROM OVER THE ARPT. THIS ACR 
DOES NOT USE ANY TERRAIN AWARENESS IN THEIR APCH BRIEFINGS. RPTR WAS COUNSELED 
REGARDING THIS. FO ADMITS TO IT BEING VERY MUCH OF A RUSHED ATMOSPHERE AND 
ADMITS TO A LOSS OF SITUATIONAL AWARENESS IN EVENT. THE FMS ON THE ACFT WAS NOT 
BEING USED. APCH PLATES WERE OUT AND USED. THE RADIO ALTIMETER WAS NOT READ 
DURING THE GPWS. ZDV KEPT FLT HIGH AND THAT STARTED THE EVENT WITH A DELAY 
VECTOR BEING NEEDED. RPTR REMEMBERS THAT INITIALLY THE CTLR WANTED TO TURN THEM 
L WHILE ON THE OVERHEAD - NE VECTOR PRIOR TO THE TURN TO THE S, BUT THIS WAS 
REFUSED ACCOUNT TSTM ACTIVITY TO THE L, N. THE TURN S WAS MADE AT A STANDARD 
RATE 25 DEG BANK ANGLE. CREW DID NOT FEEL THAT THEY WANTED TO CALL THE CTLR ON 
THE GND WHO, IT WAS THOUGHT, WAS WORKING A SPLIT POS OF APCH, TWR AND GND. 


Analysts' ratings of each assertion of relevance 
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number 

analvst 

A: similar 
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363536 

analyst 1 
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analyst 2 
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analyst 3 
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(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 7. Narrative of AS Ft S incident report number 363536, one of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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310143 

ATC CLRNCS AND HIGH TERRAIN IN SOUTH AMERICA. WE WERE FLYING INTO CALL COLUMBIA 
WHEN ATC CLRED US DIRECT TO THE CALI VOR AND DSND TO 5000 FT. CALI HAS A STAR 
AND HIGH TERRAIN OF EITHER SIDE OF THE STAR ROUTING. WHEN I CHKED THE POINTS ON 
THE STAR AGAINST OUR CURRENT DIRECT THE VOR RTE OF FLT IT LOOKED LIKE WE WOULD 
BE VERY CLOSE TO THE STAR ROUTING. WHEN YOU ARE 200 MI OUT, A 15 MI DIFFERENCE 
IS BARELY NOTICEABLE. FURTHER CHKING OF THE AREA CHART AND OUR DIRECT THE VOR 
ROUTING SHOWED TERRAIN AT 14000 FT TO 1 1000 FT DIRECTLY ALONG OUR PATH. A 
SIMILAR ATC CLRNC HAPPENS VERY OFTEN FLYING INTO LIM A, PERU. MANY, MANY, MANY 
PLTS ARE NOT AWARE OF JUST HOW CRUCIAL IT IS NOT TO ACCEPT THESE DEADLY CLRNCS. 
PLEASE GET THE WORD OUT AGAIN. 


Note: In the first sentence of the narrative, the words "CALL COLUMBIA " (referring to Cali, Colombia) are shown as 
they appear in the ASRS database. 

Analysts' ratings of each assertion of relevance 


report 

number 

analvst 

A: similar 
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310143 

analyst 1 

5 
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analyst 2 

7 

7 

7 

7 

7 

7 


analyst 3 

7 

7 

7 

2 

7 

7 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 8. Narrative of ASRS incident report number 310143, o ie of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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310228 

VERY LATE NIGHT APCH TO BOSTON LOGAN. WE DECIDED TO FLY THE APCH TO RWY 33L 
USING OUR NEW APCH NAV PROCS. THE ATIS WAS SAYING TO EXPECT THE VOR/DME GPS RWY 
33L. WE PULLED UP THIS APCH FROM OUR COMPUTER DATABASE AND INSERTED IT. UPON 
CHKING ON WITH BOS APCH WE REQUESTED THE VOR/DME RWY 33L WITH A TURN ON JUST 
OUTSIDE YARDD. APCH SAID OK AND GAVE US A HDG. WHEN TURNED ON A BASE APCH, TOLD 
US WE WERE 5 MI FROM BEEJE AND CLRED FOR THE VOR/DME GPS RWY 33L. UNBEKNOWNST 
TO US THERE ARE 2 APCHS, ONE NAMED THE (GPS) VOR/DME RWY 33L AND THE OTHER 
NAMED VOR DME OR GPS A. WE UNDERSTOOD THE CLRNC TO BE FOR THE (GPS) VOR DME RWY 
33L WHICH HAS AN INBOUND COURSE OF 342 DEGS, WHILE THE APCH CTLR UNDERSTOOD THE 
CLRNC TO BE FOR THE VOR DME OR GPS A APCH WHICH HAS AN INBOUND COURSE OF 3 10 
DEGS. SINCE WE WERE LOOKING TO INTERCEPT THE 342 DEG COURSE WE OVERSHOT THE 310 
DEG COURSE. TWR TOLD US WE WERE L OF COURSE, AT WHICH TIME WE RPTED FIELD IN 
SIGHT AND PROCEEDED FOR THE VISUAL APCH. ONE FURTHER POINT OF CONFUSION: THE 
FINAL APCH FIX FOR THE VOR DME OR GPS-A APCH IS BEEJE AND THE FAF FOR THE (GPS) 

VOR DME RWY 33L IS MEACH. THESE SOUND VERY SIMILAR AND COMBINED WITH THE ALMOST 
IDENTICAL APCH NAMES CREATES A STRONG POTENTIAL FOR CONFUSION. AT LEAST THE 
APCHS SHOULD BE RENAMED AND PROBABLY THE FAF'S AS WELL. SUPPLEMENTAL 
INFORMATION FROM ACN 310132: 1 BRIEFED AN 'APCH NAV' (RNAV) VOR/DME RWY 33L 
APCH AND INSERTED IT INTO THE FMGC. THE FO TOLD APCH THAT WE WOULD LIKE TO 
INTERCEPT THE FINAL OUTSIDE 'YARDD' (ON THE VOR/DME APCH TO RWY 33L). APCH CTL 
STATED FINE, WHATEVER WE WANTED. THIS WOULD FACILITATE CAPTURING APPR- NAV. WE 
WERE ASKED LATER ABOUT THE LENGTH OF FINAL WE DESIRED AND AGAIN REQUESTED 
'OUTSIDE OF YARDD.' THIS IS FOUND ONLY ON THE (GPS) VOR/DME RWY 33L APCH PLATE. 

APCH TURNED US ONTO A BASE LEG OF 290 DEGS AND SAID '5 MI TO BEEJE CLRED 
VOR/DME OR GPS-A 33L APCH.’ WE MISSED THE 'ALPHA' AND THE REST OF THE CLRNC 
SOUNDED LIKE WHAT WE WERE EXPECTING AND LOADED THE FMGC. YARDD IS NOT ON THE 
VOR/DME OR GPS-A APCH AND THE FAF MEACH SOUNDS A LOT LIKE BEEJE. PASSING 
THROUGH THE 3 10 DEG FINAL COURSE FOR THE VOR/DME OR GPS-A APCH TRYING TO 
INTERCEPT THE 342 DEG FINAL COURSE FOR THE VOR/DME 33L APCH, TWR TOLD US WE HAD 
FLOWN THROUGH FINAL AND SENT US BACK TO APCH. WE SUBSEQUENTLY RPTED THE ARPT 
AND RWY IN SIGHT AND WERE CLRED A VISUAL APCH. WE LANDED WITHOUT CONFLICT OR 
INCIDENT. (TOO LATE TO RELOAD ANOTHER APCH IN THE FMGC.) 


Analysts' ratings of each assertion of relevance 


report 

number 

analvst 

A: similar 
context 

B: similar 
events 

C: similar 
problems . 

D: similar 
human factors 

E: similar 
causes 

F: unspecified 
relevance 

310228 

analyst 1 

3 

3 

3 

3 

3 

2 


analyst 2 

6 

6 

6 

6 

7 

6 


analyst 3 

7 

6 

6 

6 

6 

6 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 9. Narrative of ASRS incident report number 310228, one of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts' ratings of the six assertions of relevance, as they apply to this narrative. 
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223467 

DEP LGA DEP CTL ALTERED RTE TO DIRECT COL, DIRECT WHITE AF VICE RV WHITE. FO 
ENTERED COL INTO FMS NAV SYS AND BEGAN FOLLOWING COURSE INFO. COL IN DATA BASE 
OF FMS WAS LATER FOUND TO BE COLIMA MEX VOR AND NOT COL COLTS NECK. DEP CTL 
QUESTIONED 260 DEG HDG FOR COLTS NECK. FO SELECTED #2 VOR 1 15.4 AND FOUND VOR 
NEEDLE TURNING AND 000 DME AND STARTED TURN 204 DEG TO WHITE. AGAIN DEP CTL 
QUESTIONED OUR HDG AND RECLRED US DIRECT COL- WHITE. CAPT SELECTED #1 VOR COL 
1 15.4 AND FOUND IT TO BE 125 DEG TO COL. SO FO USED #1 VOR COL 1 15.4 AND FOUND 
IT TO BE 125 DEG TO COL. SO FO USED VOR #1 AND FLEW DIRECT COL AS FILED. 

INVESTIGATION REVEALED FMS DATA BASE COL TO COLIMA MEX VOR. #2 DME INOP WITHOUT 
ANY AT FLAG. AFTER WE FOUND FO DME INDICATOR TO READ 000 DME ALL TIMES. ALL VOR 
REC AND DME ON CAPT SIDE WORK NORMALLY. UPON ARR FLL ON FLT X, I, THE CAPT, 

CALLED CHIEF PLTS OFFICE IN ATLANTA AND ADVISED THEM OF THE BEFORE MENTIONED. 

CAPT X OF THE CHIEF PLTS OFFICE CALLED ENGINEERING FOR FLT GUIDANCE. ITEM #122 
IN OUR DATA BASE WAS COL, COLIMA MEX VOR AND NOT COL FOR COLTS NECK, ABOUT 1900 
NM AND MANY MANY DEG OFF COURSE. I HAVE NOW FOUND OUT THAT MY COMPANY WILL NOT 
AND DOES NOT GUARANTEE ANY INFO IN THE FMS DATA BANK TO BE CORRECT. I MUST 
INSURE VIA THE LAT/LONG FROM A COMPANY MAP THAT THEY ARE CORRECT. THIS SYS WILL 
NOT AND DOES NOT WITH ENRTE, RTE CHANGES. SO WELCOME TO THE BACK SIDE OF THE 
ELECTRONIC WORLD, USE AT YOUR RISK OR LEAVE ALONE. WE HAVE 3 INS AND 1 FMS PER 
WDB. ABOUT 880000 DOLLARS PER PLANE. CALLBACK CONVERSATION WITH RPTR REVEALED 
THE FOLLOWING INFO. THE RPTING CAPT IN THIS INCIDENT IS RATHER EMBARRASSED 
ABOUT THE REACTION THAT HE GOT FROM HIS COMPANY WHEN HE TOLD MGMNT ABOUT HIS 
PROBLEM - THE PROBLEM BEING THAT COL (COLIMA, MEX) COMES UP ON THE FMC WHEN 
ONE ASKS FOR COL (COLTS NECK, NJ). BOTH COLS SHOULD COME UP, OFFERING THE PLT A 
CHOICE OF MEXICO OR NEW JERSEY OR ANY OTHER COL ON THE PLANET. HIS FLT MGR 
TRIED TO GET THE COMPANY ENGINEERING TO CHANGE THE DATA BASE, BUT NOTHING HAS 
BEEN DONE IN 3 WKS. NO ALTERING OR WARNING MESSAGE HAS BEEN PUT ON THE FLT 
PLANS ABOUT THIS ANOMALY. EACH CREW MUST CHK LAT/LONG AGAINST THE FLT PLAN AND 
THE COMMERCIAL CHART BEFORE DEP. THIS IS FINE, BUT WHY SPEND 1000000 
DOLLARS/ACFT, AND THEN HAVE TO DO ALL OF THIS? 


Analysts' ratings of each assertion of relevance 


report 

n , u mb&r analyst 

223467 analyst 1 
analyst 2 
analyst 3 


A: similar 
context 

B: similar 
events 

C: similar 
problems 

D: similar 
human fac tors 

E: similar 
causes 

F: unspecified 
relevance 

6 

6 

6 

6 

5 

5 

2 

6 

6 

4 

6 

6 

5 

6 

7 

6 

7 

6 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 4. Figure 10 L Narrative of AS RS incident report number 223467, one of the narratives rated as highly relevant to 
the Cali accident. Also shown are the analysts’ ratings of the six assertions of relevance, as they apply to this narrative. 
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Appendix 5. 

Examples of incidents about which the analysts disagree 




VERY LATE NIGHT APCH TO BOSTON LOGAN. WE DECIDED TO FLY THE APCH TO RWY 33L 
USING OUR NEW APCH NAV PROCS. THE ATIS WAS SAYING TO EXPECT THE VOR/DME GPS RWY 
33L WE PU LLE D UP THIS APCH FROM OUR COMPUTER DATABASE AND INSERTED IT. UPON 
CHKING ON WITH BOS APCH WE REQUESTED THE VOR/DME RWY 33L WITH A TURN ON JUST 
OUTSIDE YARDD. APCH SAED OK AND GAVE US A HDG. WHEN TURNED ON A BASE APCH, TOLD 
US WE WERE 5 MI FROM BEEJE AND CLRED FOR THE VOR/DME GPS RWY 33L. UNBEKNOWNST 
TO US THERE ARE 2 APCHS, ONE NAMED THE (GPS) VOR/DME RWY 33L AND THE OTHER 
NAMED VOR DME OR GPS A. WE UNDERSTOOD THE CLRNC TO BE FOR THE (GPS) VOR DME RWY 
33L WHICH HAS AN INBOUND COURSE OF 342 DEGS, WHILE THE APCH CTLR UNDERSTOOD THE 
CLRNC TO BE FOR THE VOR DME OR GPS A APCH WHICH HAS AN INBOUND COURSE OF 310 
DEGS. SINCE WE WERE LOOKING TO INTERCEPT THE 342 DEG COURSE WE OVERSHOT THE 3 10 
DEG COURSE. TWR TOLD US WE WERE L OF COURSE, AT WHICH TIME WE RPTED FIELD IN 
SIGHT AND PROCEEDED FOR THE VISUAL APCH. ONE FURTHER POINT OF CONFUSION: THE 
FINAL APCH FIX FOR THE VOR DME OR GPS-A APCH IS BEEJE AND THE FAF FOR THE (GPS) 

VOR DME RWY 33L IS MEACH. THESE SOUND VERY SIMILAR AND COMBINED WITH THE ALMOST 
IDENTICAL APCH NAMES CREATES A STRONG POTENTIAL FOR CONFUSION. AT LEAST THE 
APCHS SHOULD BE RENAMED AND PROBABLY THE FAFS AS WELL. SUPPLEMENTAL 
INFORMATION FROM ACN 310132: 1 BRIEFED AN 'APCH NAV' (RNAV) VOR/DME RWY 33L 
APCH AND INSERTED IT INTO THE FMGC. THE FO TOLD APCH THAT WE WOULD LIKE TO 
INTERCEPT THE FINAL OUTSIDE 'YARDD' (ON THE VOR/DME APCH TO RWY 33L). APCH CTL 
STATED FINE, WHATEVER WE WANTED. THIS WOULD FACILITATE CAPTURING APPR- NAV. WE 
WERE ASKED LATER ABOUT THE LENGTH OF FINAL WE DESIRED AND AGAIN REQUESTED 
'OUTSIDE OF YARDD.' THIS IS FOUND ONLY ON THE (GPS) VOR/DME RWY 33L APCH PLATE. 

APCH TURNED US ONTO A BASE LEG OF 290 DEGS AND SAID '5 MI TO BEEJE CLRED 
VOR/DME OR GPS-A 33L APCH.' WE MISSED THE 'ALPHA' AND THE REST OF THE CLRNC 
SOUNDED LIKE WHAT WE WERE EXPECTING AND LOADED THE FMGC. YARDD IS NOT ON THE 
VOR/DME OR GPS-A APCH AND THE FAF MEACH SOUNDS A LOT LIKE BEEJE. PASSING 
THROUGH THE 310 DEG FINAL COURSE FOR THE VOR/DME OR GPS-A APCH TRYING TO 
INTERCEPT THE 342 DEG FINAL COURSE FOR THE VOR/DME 33L APCH, TWR TOLD US WE HAD 
FLOWN THROUGH FINAL AND SENT US BACK TO APCH. WE SUBSEQUENTLY RPTED THE ARPT 
AND RWY IN SIGHT AND WERE CLRED A VISUAL APCH. WE LANDED WITHOUT CONFLICT OR 
INCIDENT. (TOO LATE TO RELOAD ANOTHER APCH IN THE FMGC.) 


Analysts' ratings of each assertion of relevance 


report 

number 

analyst 

A; similar 
context 

B: similar 
events 

310228 

analyst 1 

3 

3 


analyst 2 

6 
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analyst 3 

7 
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C: similar 

D: similar 

E: similar 

problems 

human factors 

causes 

3 

3 

3 

6 

6 

7 

6 

6 

6 


F: unspecified 
relevance 
2 
6 
6 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 5. Figure 1. Narrative of AS RS incident report number 310228, one of the narratives rated as relevant to the 
Cali accident by two of the analysts, but not rated as relevant by the other analyst. Also shown are the analysts' ratings of 
the six assertions of relevance, as they apply to this narrative. 
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140711 

AFTER PASSING KILER (N15:00.0 W 76:52.0), BARRANQUILLA CTL CLRED US DIRECT 
TULUA VOR. AS WE WERE PASSING ABEAM CARTAGENA VOR, AN LGT Y CROSSED OUR NOSE 
HDG INANE DIRECTION. IT WAS EXTREMELY CLOSE AND WE ARE SURE HE WAS AT OUR ALT 
BECAUSE WE HIT HIS WAKE TURBULENCE AS WE PASSED BEHIND HIM. BARRANQUILLA CTL 
STATED THAT HE WAS CTLING NO OTHER ACFT IN OUR AREA AT FL330. IN THE FUTURE, I 
WILL NOT ACCEPT AN OFF AIRWAYS CLRNC WHEN NOT POSITIVE OF BEING IN RADAR 
CONTACT. CTLR STATED THAT SOMETIMES ACFT TRANSVERSE HIS AIRSPACE THAT HE WAS 
NOT CTLING. ALSO, BARRANQUILLA RADIO VERY WEAK AND UNCLEAR. I NORMALLY FLY WITH 
THE RWY TURNOFF LIGHTS ON AT ALL TIMES AND WAS ONCE CRITICIZED BY A COMPANY 
CHECK PLT FOR THIS PRACTICE. ON THIS PARTICULAR DAY, MY RIGHT TURNOFF BURNED 
OUT AND POPPED ITS CIRCUIT BREAKER. THE CONFLICTING TFC CAME FROM THE DIRECTION 
OF THE SETTING SUN SO HIS CHANCES OF SEEING US WAS MUCH GREATER THAN US SEEING 
HIM. AFTER I MADE MY NEAR MISS REPORT TO BARRANQUILLA CTL, THERE WAS A LOT OF 
JABBERING IN SPANISH BETWEEN BARRANQUILLA AND OTHER ACFT. I BELIEVE SOMEONE IS 
COVERING UP A MISTAKE. WE HAD BEEN ASKED TO SQUAWK A XPONDER CODE LONG BEFORE 
THE NEAR MISS SO WE ASSUMED WE WERE BEING PROVIDED RADAR SEPARATION. NOW, I’M 
NOT SO SURE. MY SUGGESTION TO OTHER PLTS WHO FLY SOB (SOUTH OF THE BORDER) IS 
TO USE ANY MEANS AVAILABLE TO AVOID A SIMILAR SITUATION. SOME SUGGESTIONS ARE 
USE OF EXTERIOR LIGHTS, FLYING 100' ABOVE OR BELOW ASSIGNED FLT LEVEL, NOT 
FLYING OFF AIRWAY DIRECT ROUTES, AND OF COURSE EXTREME VIGILANCE. SUPPLEMENTAL 
INFORMATION FROM ACN 140497: BAQ CTL HAD NO KNOWLEDGE OF THE CONFLICTING ACFT, 
ONLY REPORTED TFC WAS AT FL3 10. 


Analysts' ratings of each assertion of relevance 


report 

number 

analyst 

A: similar 
context 

B: similar 
events 

C: similar 
problems 
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human factors 

E: similar 
causes 
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relevance 

140711 

analyst 1 
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5 
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analyst 2 
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1 

1 
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analyst 3 

5 

5 

6 
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6 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 5. Figure 2. Narrative of ASRS incident report number 14071 1 , one of the narratives rated as relevant to the 
Caii accident by two of the analysts > but not rated as relevant by the other analyst Also shown are the analysts' ratings of 
the six assertions of relevance, as they apply to this narrative . 
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153355 

F/0 WAS FLYING THE CHINS ONE ARR INTO SEATAC USING THE FMS FOR PRIMARY NAV. THE 
FMS DISPLAY WAS SELECTED AND SHOWED US TO BE ON COURSE. BOTH THE B AUTOPLT WITH 
FMS SELECTED IN THE NAV MODE AND AUTOTHROTTLES WERE SELECTED AND ENGAGED. 
APPROX 40 NM +/- SE OF SEA VOR, APCH CTL TOLD US THAT HIS RADAR SHOWED US TO BE 
+/- 3 MI N OF COURSE. APCH CTL GAVE US A L TURN VECTOR TO RETURN TO COURSE. A 
CHK OF THE SEA VOR R SHOWED THAT EVEN THOUGH FMS SHOWED CTRED ON COURSE IT WAS 
IN ERROR. APCH CTL DID NOT REQUEST THAT WE INTERCEPT THE ARR AND THE XED 
THROUGH THE ARR R (SEA 101 DEG). APCH CTL THEN GAVE US A R TURN VECTOR AND WE 
BELIEVED THAT WE WERE THEN NOT ON THE PUBLISHED ARR. IN FACT THE F/O REMARKED 
THAT WE MUST BE IN AN APCH CTL VECTOR FOR A W DOWNWIND FOR A S LNDG AT SEATAC. 
THEREFORE, WE DID NOT MAKE THE PUBLISHED TURN TO 340 DEG. SUBSEQUENT COMMENTS 
FROM APCH SUGGESTED THAT THIS WAS NOT UNCOMMON INDICATING PROBS WITH THIS ARR. 


Analysts' ratings of each assertion of relevance 


report 

number 

analyst 

A: similar 
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B: similar 
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C: similar 
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causes 
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153355 

analyst 1 

6 
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analyst 2 
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5 


analyst 3 

3 

3 

2 
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2 


(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 5. Figure 3. Narrative of AS RS incident report number 153355, one of the narratives rated as relevant to the 
Cali accident by two of the analysts, but not rated as relevant by the other analyst. Also shown are the analysts' ratings of 
the six assertions of relevance, as they apply to this narrative. 
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355364 

ON DSCNT TO 3200 FT MSL (MV A) AT APPROX 3400 FT, WE RECEIVED A GPWS WARNING 
TOO LOW, TERRAIN.' FLC IMMEDIATELY PERFORMED ESCAPE MANEUVER. AT APPROX 3800- 
3900 FT, AND IN VMC, WE STARTED OUR DSCNT BACK DOWN TO 3200 FT AND INFORMED 
APCH CTL OF OUR SIT. ACFT RATE OF DSCNT WAS APPROX 700 FPM. WE BELIEVE THAT 
EVEN THOUGH WE WERE ABOVE MVA, THE COMBINATION OF RISING TERRAIN AND ACFT DSCNT 
RATE MAY HAVE CAUSED THE GPWS TO GIVE ITS WARNING. RADAR ALT WAS 1600-1800 FT 
AGL AT TIME OF WARNING. 


Analysts’ ratings of each assertion of relevance 
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analyst 1 
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analyst 2 
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analyst 3 
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(1: strongly disagree, 2: disagree, 3: somewhat disagree, 4: undecided, 5: somewhat agree, 6: agree, 7: strongly agree) 


Appendix 5. Figure 4. Narrative of ASRS incident report number 355364, one of the narratives rated as irrelevant to the 
Cali accident by two of the analysts, but rated as relevant by the other analyst. Also shown are the analysts ' ratings of the 
six assertions of relevance, as they apply to this narrative. 
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Appendix 6. 

Excerpts of each of the 84 relevant incidents, 
categorized by prominent features they share with the Cali accident 




Appendix 6. Excerpts of each of the 84 relevant incidents, categorized by prominent features they share with the Cali 
accident. In summary, among the 84 incidents rated as relevant by QUORUM and also rated as relevant by one or more of 
the analysts: 37 incidents involved over-reliance on automation, and other problems with the use of automation; 29 
incidents involved confusion, changes, and problems during descent/approach; 19 incidents involved terrain avoidance; 
and 6 incidents involved operations in foreign airspace. The number of incidents under each heading appears in 
parentheses at the end of each header. The numbers sum to 91 because five of the incidents appear in two different 
places, and another one appears in three places. Because of these and other cases of categorical overlap, the numbers 
in the headings should not be over-interpreted. Clearly, the categories cannot be logically distinct because the use of 
automation and the occurrence of incidents during descent/approach are pervasive, and other features are also shared. 
Accordingly, assignments to categories are based on the more prominent features of the incidents that are shared with 
the Cali accident. As used here, the term ", automation ” refers to the FMS (flight management system) or other 
components of the automated flight systems that are used to operate the aircraft. 

1. Over-reliance on automation, and other problems wi th use of automation (37) 

1.1. Over-reliance on automation (36) 

Incidents involving over-reliance on automation typically also involved course or altitude deviations. Such deviations imply a 
loss of situational awareness on the part of the crew. 

1.1.1. Automation turns aircraft off course (5) 


Incidents 223467 and 224363 not only involved automation turning the aircraft off course, but also involved confusion of 
identical names in the FMS leading to course deviations. 


223467 DEP LGA DEP CTL ALTERED RTE TO DIRECT COL...FO ENTERED COL INTO FMS NAV SYS AND BEGAN 
FOLLOWING COURSE INFO. COL IN DATA BASE OF FMS WAS LATER FOUND TO BE COLIMA MEX [i.e., Mexico] VOR AND 
NOT COL COLTS NECK [New Jersey], DEP CTL QUESTIONED 260 DEG HDG FOR COLTS NECK....I HAVE NOW FOUND OUT 
THAT MY COMPANY WILL NOT AND DOES NOT GUARANTEE ANY INFO IN THE FMS DATA BANK TO BE CORRECT I MUST 
INSURE VIA THE LAT/LONG FROM A COMPANY MAP THAT THEY ARE CORRECT, [also see 223467 in "1.1.4. Name 
confusion using automation"] 


224363 UPON DEP, LGA DEP CTL ALTERED RUNG TO DIRECT COL...FO ENTERED COL IDENTIFIER INTO FMS NAV SYS 
AND BEGAN FOLLOWING COURSE INFO. COL IN DATA BANK OF FMS WAS LATER VERIFIED 1900 MI FROM COL. DEP 
CTLR QUESTIONED 260 DEG HDG FOR COLTS NECK....THIS RFT SEEMS TO BE A MATCH OF A PREVIOUS PROBLEM WITH 
COLTS NECK/COLIMA HAVING THE SAME IDENTIFIER, COL THE RPTR IN THIS INCIDENT STATES THAT THE ANOMALY IS 
STILL IN THE DATA BASE... THIS RPTR STATES THAT THERE ARE OTHER HOLES IN THE DATA BASE, SPECIFICALLY 
INCLUDING THE NANCI 4 ARR INTO LGA AND THE DARBS 1 ARR INTO TPA. HE PROMISES TO CONTINUE TO FIGHT TO 
IMPROVE THE DATA BASE, [also see 224363 in "1.1.4. Name confusion using automation”] 


Incident 146645 not only involved automation turning the aircraft off course, but also involved effects of changes during 
descent/approach, confusion regarding charts, and erroneous data entry. 


146645 I TUNED AND SET THE NAV EQUIP FOR THE ILS TO RWY 32 AND ANNOUNCED THAT THE APCHCHK WAS 

COMPLETE... HOWEVER, OUR POS AND HDG RELATIVE TO THE LOC WAS LOOKING LIKE AN EXTREMELY TIGHT TURN 
ON OR AN OVERSHOOT, SO I REQUESTED OUR CURRENT HDG AND ALT ASSIGNMENT FROM APCH. WHAT WE RECEIVED 
WAS A TURN AND CLRNC FOR THE NDB APCH RWY 30 CIRCLE TO LAND RWY 32 AT STP AND TO CONTACT ST PAUL 
TWR. WHILE HURRYING TO GET THE NAV EQUIP RETUNED AND IDENTED AND RECONFIGURING THE ACFT I WENT 
RIGHT BY THE NDB RWY 30 CHART AND TOOK THE DATA FROM THE NDB RWY 3 ST PAUL, MN, LAKE ELMO. AS WE 
WENT BY THE NDB AND STARTED TO TURN TO THE DMBND HDG, SOME 70 DEG FROM OUR CURRENT HDG, WE KNEW 
SOMETHING WAS WRONG. A QUICK CHK OF THE CHART CONFIRMED THE ERROR I HAD MA DE ... T HE TWR CALLED 
AND SAID APCH SHOWED US NE OF COURSE ... WE, AS A FLT CREW, SHOULD NOT HAVE ACCEPTED THE CLRNC FOR 

AN APCH TO A RWY DIFFERENT FROM THE ONE WE WERE SET UP FOR, UNLESS WE WERE SURE WE HAD TIME TO SET UP 

AND CONFIGURE FOR THE APCH TO THE NEW RWY. [also see 146645 in "2.2.2. Confusion regarding charts" and "2.1 . Last 
minute approach/runway change leads to significant confusion"] 


Incident 347848 not only involved automation turning the aircraft off course, but also involved loss of data when other data are 
entered. 

347848 CAPT ... WAS OCCU PIED IN COMPLETING APCH SETUP IN FMS... . SEQUENCE IN FMS PROGRAMMING WAS 
SELECTING RWY 25L, LNAV DIRECT DOWNE, THEN BACK TO DEP/ARR PAGE TO RESELECT RWY 25L, CIVET, ARNES 
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TRANSITION. THIS DROPPED DOWNE AS ACTIVE WAYPOINT, AND START! D JET IN TURN BACK TO ARNES, [also see 
347848 in "1.1.2. Loss of data when other data are entered "] 

353338 THE CAFT HAD BRIEFED A VISUAL APCHTORWY 36L WITH AN RNAV RWY 36L APCH BACKUP. HE HAD 
ENTERED THE RNAV APCH INTO OUR FM S AND COUPLED THE FMSTO THE AUTOPLT WHICH WAS FLYING THE ACFT . 
WHEN WE CROSSED THE LEESE FIX THE AUTOPLT TURNED R AND HEADED FOR THE 'CAMBE INTXN WHICH WAS THE 
INITIAL APCH FIX FOR THE RNAV RWY 36L APCH. ... THE FMS, COUPLED TO THE AUTOPLT IN THE NAV MODE, WAS 
DOING EXACTLY AS THE CAFT HAD PROGRAMMED IT, SO HE DID NOT SUSPECT A PROB. ... CTLR ASKED WHERE DID 
YOU GET THAT HDG? ... THE CTLR HAD EXPECTED THEM TO CONTINUE ON TO THE VOR AS SHOWN ON THE ARR CHART 

1.1.2. Loss of data when other data are entered (5) 

272508 WHAT I FAILED TO NOTICE WAS THAT BY INSERTING THE ARR IN THE FMS, THE COMPUTER DUMPED THE 
XING RESTRICTION I HAD INSERTED JUST A FEW MOMENTS EARLIER. ... THE CAUSE, I BELIEVE, WAS A COMBINATION 
OF COCKPIT MGMNT OVERLOAD DURING THE APCH PHASE COUPLED WITH AN OVERCONFIDENCE IN THE FMS TO 
PRESENT VALID DSCNT PROFILE INFO. I ALLOWED MYSELF TO GET TOO BUSY DURING THE DSCNT TO MAKE 
ESSENTIAL XCHKS TO CONFIRM THE FMS WAS WORKING AS ADVERTISED. 

279030 IN ENTERING THE [ILS approach] DATA INTO THE DATABASE, WE WERE UNAWARE THAT THE 15000 FT 
CONSTRAINT AT MUMSY WAS DELETED. THEREFORE, THE ACFT HAD REVERTED TO A 1000 FPM RATE OF DSCNT. WE 
REALIZED THE ERROR AS WE WERE XING OVER MUMSY AT ABOUT 18000 FT. 

361956 DURING DSCNT TO BRONC INTXN .. I ENTERED THE LNDG RWY 33L INTO THE FMS. THIS CAUSED THE RESTR 
AT BRONC (280 KTS/1 1000 FT) TO DELETE ITSELF. I HAD THE ACFT SELECTED TO MANAGED FLT AND DID NOT 
OBSERVE THAT IT CHANGED TO VERT SPD AND THE RESTR DELETED. CTR REALIZED THAT WE PROBABLY WOULD NOT 
MAKE THE RESTR AND GAVE US A VECTOR. ... THIS WOULD NOT OCCUR IF ONCE A RESTR WAS PLT ENTERED INTO THE 
FMS IT WOULD NOT AUTOMATICALLY DELETE ITSELF WHEN A DEST RWY IS ENTERED. 


Incident 275413 also involved problems due to changes late in descent/approach. 


275413 AT THIS POINT, TO COMPLY WITH THE 210 K1AS SPD REQUEST BY ATC, THE FO SELECTED THE VERT SPD' 
FUNCTION ON THE DIGITAL FLT GUIDANCE PANEL, SETTING 600 FPM DSCOT RATE AND CHANGING THE SPD SELECT 
WINDOW TO SHOW 210 KTS. BECAUSE OF A FLAW IN THE MD-88'S DIGITAL FLT GUIDANCE PROGRAM, LAST SECOND 
CHANGES IN ANY OF THE VERT CTL FUNCTIONS WILL REMOVE THE ALT LEVEL OFF COMMAND PREVIOUSLY 
SELECTED. THE ALT WARNING SIGNAL AND A TCASD TA' GOING OFF SIMULTANEOUSLY AT 7700 MSL BROUGHT OUR 
ATTN TO THE DEV. [also see 275413 in "2.4. Other problems with changes la-.e in descent/approach"] 


Incident 347848 also involved automation turning the aircraft off course. 


347848 CAPT ... WAS OCCUPIED IN COMPLETING APCH SETUP IN FMS... . SEQUENCE IN FMS PROGRAMMING WAS 
SELECTING RWY 25L, LNAV DIRECT DOWNE, THEN BACK TO DEP/ARR PAGE TO RESELECT RWY 25L, CIVET, ARNES 
TRANSITION. THIS DROPPED DOWNE AS ACTIVE WAYPOINT, AND STARTED JET IN TURN BACK TO ARNES, [also see 
347848 in "1.1.1. Automation turns aircraft off course"] 


1.1.3. Distracted by automation (7) 


Incident 368360 is one of the two incidents rated by all three analysts as highly relevant to the Cali accident. The other is 
310130 in section 2.1. See appendix 3, table 1 for the list of incidents sorted on their relevance ratings. 


368360 WE WERE AGAIN GIVEN INVALID ROUTING (RTE BREAK ON FMS) AT THIS POINT MY FO BECAME ENGROSSED 
IN LOOKING ON OUR HIGH AND LOW ALT ENRTE CHARTS TO FIND WHERE THE PROB WAS. I TRIED AGAIN TO ENTER 
ROUTING ON THE FMS (THIS TOOK APPROX 2 MINS). I LOOKED UP TO SEE MY FLT INSTS AND AT THIS TIME NOTED THE 
ALTIMETER READING FL3 12 AND CUBING. I IMMEDIATELY DISCONNECTED THE AUTOPLT... APPROX 5 SECONDS 
LATER ZB W TOLD US TO MAINTAIN FL3 1 0. ... I BELIEVE THAT THE COMPLEXITY OF FMS PROGRAMMING IS NOT 
ADDRESSED IN INITIAL TRAINING AT SCHOOL BECAUSE EACH ACFT HAS DIFFERENT EQUIP. HOWEVER, THIS LEAVES 
THE FLC TO LEARN AS THEY FLY.’ THIS EFFECTIVELY TOOK MY FO OUT O- THE LOOP IN THAT IF HE WAS 
PROGRAMMING THE FMS, I COULD HAVE CONCENTRATED MORE ON MOf STORING THE ACFT... THE ENTIRE CREW WAS 
DISTR, AND WE BOTH FAILED TO MONITOR THE PERFORMANCE OF THE ACFT. 

359641 WE WERE NAVING USING FMS. WORKLOAD WAS HIGH FOR BOTH PLTS, AND THE FMS DID NOT APPEAR TO BE 
INTERCEPTING THE COURSE. NEITHER PLT NOTICED THE ALT PASSING 5(00 FT. AT 5300 FT, THE ALT ALERTER 
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SOUNDED... THE MAJOR CAUSE OF THIS DEV WAS WHEN BOTH PLTS WERE DISTRACTED BY A MINOR FMS PROB 
DURING A BUSY DEP. THE PF SHOULD HAVE IGNORED THE FMS AND SWITCHED TO RAW DATA NAV. 

160843 I CALLED THE APCH UP ON THE FMS AND REALIZED THAT THE APCH IN THE FMS DATA BASE DID NOT GIVE 
ANY PRECISE GUIDANCE TO THE APCH END OF THE RWY. I STARTED TO BUILD THE APCH... THE PREOCCUPATION OF 
BOTH CREW MEMBERS CAUSED A DELAY IN COMPLETION OF THE DSNT CHKLIST (USUALLY DONE PASSING THROUGH 
1 8000 ) UNTIL JUST AFTER LEVEL OFF AT 1 1 000'. WE RESET THE ALTIMETERS TO 30.22 AND REALIZED WE HAD 
LEVELED OFF AT 1 1 300'. 

219222 THE FO DID THE REQUIRED FMS ENTRIES. ENCOUNTERING DIFFICULTY WITH THE FORMAT FOR ENTRY MY 
ATTN WAS DIVERTED TO EXPLAIN THE FORMAT FOR FMS TO THE FO. UPON COMPLETION OF THE ENTRY THE DSCNT 
INFO WAS SLOW BEING DISPLAYED (AN UNFORTUNATE CHARACTERISTIC OF THE WDB ACFT FMS). MEANWHILE, 
MENTAL CALCULATIONS INDICATED THE XING RESTRICTION COULD NOT BE MADE.. MY OWN INEXPERIENCE WITH 
THIS ACFT SIMPLY DID NOT ALERT ME TO THE DIFFICULTY IN COMPLYING WITH THE CLRNC WHEN FIRST ISSUED AND 
THE FMS FORMAT PROBLEM ENCOUNTERED BY THE FO ARE ALL TOO COMMON ON THIS ACFT... . 

184380 I SHOULD HAVE MONITORED MORE CLOSELY ON HOW THE CAPT HAD LOADED THE FMS ON ARR. AFTER I 
FOUND THE DISCREPANCIES I BECAME OVERLOADED ON KEEPING UP ON WHAT THE CAPT WAS DOING AND WHAT WAS 
NEEDED TO CORRECTLY FLY THE APCH AND DO ALL THE CHKLIST ITEMS. 

296506 WE LANDED ON RWY 9R. ... WE HAD NOT RECEIVED LNDG CLRNC FROM ATLANTA TWR. THIS EVENT COULD 
HAVE POSSIBLY BEEN AVOIDED DF THE CAPT HAD NOT BEEN PROGRAMMING THE FMS DURING THE APCH. 

350190 AUTOPLT ENGAGED, THE ACFT ENTERED A POCKET OF SEVERE TURB. ... FO APPROPRIATELY DISCONNECTED 
AND MADE PROPER INPUTS MANUALLY. ... ONCE WE STABILIZED AT 16000 FT, I WAS THEN REQUIRED TO MAKE 
AMENDMENTS TO THE FMS FOR THE APCH AND BRIEF THE APCH. DUE TO THESE DESCRIBED FACTORS, I FORGOT TO 
RFT THE TURB. 

1.1.4. Name confusion using automation (4) 

(Also see section "2.2.1. Name confusion" in section "2.2. Other confusion during descent/approach.") 


The Cali accident involved two distinct instances of name confusion. First, the arrival was named ROZO 1, instead of TULUA 1, 
which appeared to confuse the captain. According to the NTSB report (NTSB, 1996c), "CVR (cockpit voice recorder) evidence 
reveals that the crew may have expected the standard STAR naming convention to be used with respect to the ROZO 1 Arrival and 
may have incorrectly believed that ROZO was located at the beginning of the route." The second instance of name confusion 
involved crew confusion of the identifier R for ROZO, which the automated system interpreted to mean ROMEO. 


Incidents 223467 and 224363 both involved confusion of identical names in the FMS leading to course deviations. 

223467 DEP LGA DEP CTL ALTERED RTE TO DIRECT COL...FO ENTERED COL INTO FMS NAV SYS AND BEGAN 
FOLLOWING COURSE INFO. COL IN DATA BASE OF FMS WAS LATER FOUND TO BE COLIMA MEX [i.e., Mexico] VOR AND 
NOT COL COLTS NECK [New Jersey], DEP CTL QUESTIONED 260 DEG HDG FOR COLTS NECK....I HAVE NOW FOUND OUT 
THAT MY COMPANY WILL NOT AND DOES NOT GUARANTEE ANY INFO IN THE FMS DATA BANK TO BE CORRECT. I MUST 
INSURE VIA THE LAT/LONG FROM A COMPANY MAP THAT THEY ARE CORRECT, [also see 223467 in ”1 .1.1. Automation 
turns aircraft off course"] 


224363 UPON DEP, LGA DEP CTL ALTERED RTING TO DIRECT COL...FO ENTERED COL IDENTIFIER INTO FMS NAV SYS 
AND BEGAN FOLLOWING COURSE INFO. COL IN DATA BANK OF FMS WAS LATER VERIFIED 1900 MI FROM COL. DEP 
CTLR QUESTIONED 260 DEG HDG FOR COLTS NECK... .THIS RPT SEEMS TO BE A MATCH OF A PREVIOUS PROBLEM WITH 
COLTS NECK/COLIMA HAVING THE SAME IDENTIFIER, COL THE RPTR IN THIS INCIDENT STATES THAT THE ANOMALY IS 
STILL IN THE DATA BASE....THIS RPTR STATES THAT THERE ARE OTHER HOLES IN THE DATA BASE, SPECIFICALLY 
INCLUDING THE NANCI 4 ARR INTO LGA AND THE DARBS 1 ARR INTO TPA. HE PROMISES TO CONTINUE TO FIGHT TO 
IMPROVE THE DATA BASE, [also see 224363 in "1.1.1. Automation turns aircraft off course"] 

315261 AT THIS POINT I REALIZED THAT I HAD BEEN FLYING DIRECT TO THE OM (FONT A) INSTEAD OFPETIS NDB. 

THE FMS NAV DATA BASE LISTS THE OM (FONTA) AS FF26L INSTEAD OF JUST FONTA. MOST NDB'S ARE COLLOCATED 
WITH THE OM, REFERRED TO AS LOM'S. HUMAN FACTORS BEING WHAT THEY ARE, I SAW FF26L AND USED THAT 
WAYPOINT THINKING AT THAT MOMENT THEY WERE COLLOCATED AND I WAS FLYING TO PETiS. ACTUALLY THEY ARE 
ABOUT 5 NM APART. DF THE FMS HAD SHOWN FONTA AS THE OM WAYPOINT INSTEAD OF FF26L I THINK IT WOULD 
HAVE BEEN CLR THAT THEY ARE IN FACT 2 DIFFERENT POINTS. THERE DOES NT SEEM TO BE ANY CONSISTENCY WITH 
THE FMS APCH DATA BASE. SOME APCHS SHOW THE OM NAME AND OTHERS USE THE FF (FINAL FIX) FORMAT. THEY 


69 



SHOULD ALL USE THE CORRECT FIX NAMES. ADDING TO THE CONFUSION THE APCH DATABASE SHOWS PETIS AS 
SBNB. 

301760 I DISCOVERED THAT MRLIN WAS NOW ABOUT 5 MI N OF ITS PREVIOUS LOCATION AND TRITN WAS WHERE 
MRLIN USED TO BE OUR PROB ACTUALLY OCCURRED BECAUSE WE FAILED TO RECOGNIZE THAT THE ACTIVE DATA 
BASE IN OUR FMS WAS NOT THE CURRENT DATA BASE DURING THE FMS PREFLT. PERHAPS WHEN THEY ARE 
CONSIDERING MOVING THE LOCATION OF A FIX THEY SHOULD CONSIDER CHANGING THE NAME AS WELL. 


1.1.5. Automation data entry error or data error (6) 


321 136 AFTER THE FMS RAN OUT ALL THE LEGS, AND WE WERE NOT WITHIN RANGE OF LAND, I DECLARED AN EMER 
DUE TO OUR INABILITY TO DETERMINE OUR POS. ... FO SAYS LAST MINUTE CONFUSION WHEN DEPARTING RJNN 
CONTRIBUTED TO THE CAPT INITIALIZING PRESENT POS INCORRECTLY. INSERTED WRONG LONGITUDE. ... THE 
LONGITUDE WAS E AND SHOULD HAVE BEEN W. ... FUEL WAS DANGEROUSLY LOW, THE FLC HAD NO NAV TO THE 
HAWAIIAN ISLANDS AND DECLARED AN EMER. COMMERCIAL RADIO RECOMMENDED A HDG CHANGE AND WHEN 
FINALLY RECEIVING AN ADF SIGNAL, IT TOOK A 90 DEG TURN TO THE R. 'PHEY HAD BEEN PARALLELING THE CORRECT 
COURSE, BUT 350 MI OFF. 

307543 ACFT WAS ON AUTOPLT USING FMS NAV IN MD-1 1 ACFT. APCH CTL DIRECTED ACFT ON A HDG TO INTERCEPT 
THE RWY 5L LOC AT RCPT. ... APCH ALERTED CREW TO BEING SE OF COURSE. ... LOC NEEDLE SHOWED CTRED BUT 
LOC FREQ HAD AUTO-TUNED TO ICKS, THE LOC FREQ FOR ILS-DME RWY 6, WITHOUT THE PLTS KNOWLEDGE NOR 
THROUGH EITHER PLTS ACTIONS. RECEIVED VECTOR TO INTERCEPT RWY 5L LOC ... HE DOES NOT KNOW IF THE CAPT 
RETUNED THE ILS MANUALLY , AND PUT IN THE WRONG ELS , OR WHETHER THE FMS JUMPED TO THE WRONG DLS BY 
ITSELF. ... THE RPTENG FO KNOWS THAT HE DID NOT AURALLY CHK THE ELS IDENTIFIER AND HE BELIEVES THAT THE 
CAPT PROBABLY DID NOT EITHER. 

342838 DEP CTL SHOWED US OFF COURSE FROM THE SEA 158 DEG RADIAL. I STATED THAT WE WERE CURED VIA THE 
FMS DEP AND WERE FOLLOWING THE COURSE IN FMS LNAV AND VNAV. CTLR THEN CANCELED OUR FMS DEP AND 
ISSUED RADAR VECTORS TO INTERCEPT THE AIRWAY. CTLR THEN STATI3D, DONT USE THAT FMS DEP, IF WE WERE ON 
N DEPS YOU COULD GET KILLED DOING THAT.’... THE RPTR STATED THAI PRIOR TO DEP HE DID NOT NOTICE IF THE IRS 
WAS ALIGNED. 

358123 THE HDG BUG IMMEDIATELY SLEWED TO A COURSE OTHER THAN THE INBOUND COURSE AND THE COMMAND 
BARS WERE COMMANDING TURN TO THE HEX3 BUG. THE CAPT COCKED OFF THE AUTOFLT SYS AND FLEW AN 
UNEVENTFUL MANUAL RAW-DATA APCH. ... WHILE BRIEFING, THE CAPTS BRIEF WAS INTERRUPTED SEVERAL TIMES 
WITH ATC ALT CHANGES, TFC CALLS AND A FREQ CHANGE THE INTERRUPTIONS PROBABLY CAUSED US TO NOT 
PROPERLY VERIFY THE COURSE IN THE ILS CTL PANEL. 

302770 THIS WAS A CASE OF THE FMS PROVIDING FAULTY DSCNT DATA TO THE FLC AND THE FLC FAILING TO BACK 
UP THE DATA. RELYING ON THE FMS TO SHOW THE DSCNT CAUSED US TO BE TOO HIGH TO CROSS A PUBLISHED RESTR 


156414 WE WERE NAVIGATING USING THE FMS... ATC CALI ED... BECAUSE HE SHOWED US ABOUT 8 NM N OF COURSE. 
...EVIDENTLY FMS SYS WAS WORKING WITH ERRONEOUS XWIND OF 35CV101. WHEN WE SELECTED ’’PROGRESS PAGE 
OF FMS IT SHOWED NO NAV AIDS IN USE, THE SYS WAS ’’LOST '. WE RESET OUR PRESENT POS (USING NAV AID AND 
DME) ON FMS POS PAGE. ...IT CAUGHT US OFF OUR GUARD. 


1.1.6. Other problems getting automation to work as desired (4) 


341815 I INFORMED THE CREW TO RPT A DUAL IRS NAV FAILURE AS THEY DID NOT AGREE WITH THE CTLR . . AT OUR 
DEST, THE FO DISCOVERED THE PROB. THE VERY DANGEROUS ERROR AND HIDDEN HONEYWELL TRAP.’ WE ENTERED 
AGNEZ MANUALLY AS IT WAS NOT DEFINED IN THE DATABASE IN SO DOING, THE FO GAVE ME THE WRONG DISTANCE 
FOR PLACE-BEARING-DISTANCE.. I WAS UNABLE TO CORRECT THE ERROR TO MY AMAZEMENT GETTING ERRORS FIX 
USED BY ACTIVE FLT PATH.[’] NOT TO WORRY, I DELETED THE FIX FROM THE ACTIVE FLT PATH AND REENTERED IT, 
ONLY I FOUND I WAS UNABLE TO NOW DELETE AND CORRECT IT FOR REASONS UNDETERMINED... THE FLT WAS 
COMPLETED... [On a subsequent flight,] THE INCORRECT FIX WITH CORRECT PLACE-BEARING AND WRONG DISTANCE 
WAS STILL IN THE DATABASE ... THE HONEYWELL TRAP! ! ! THERE IS NO WAY HONEYWELL MAKES THE PLT AWARE 
THAT THE FIX HE CHOSE WAS MANMADE AND NOT FROM THE DATABASE! 

117306 WE GLANCED AT OUR CHARTS, LOCATED POPPS ON THEM, AND THE CAPT TRIED TO ENTER IT AS A WAYPOINT 
IN THE FMS. THE FMS REJECTED IT AS "NOT IN DATA BASE”. ... AT THAT POINT, THE CTLR ASKED IF WE KNEW WE 
WERE ”5 MILES” PAST POPPS. ... OUR FMS DATA BASE SHOULD INCLUDE MOST, IF NOT ALL, POTENTIAL HOLDING 
FIXES NEAR ARPTS. 
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1 16871 CAPT ATTEMPTED TO PROGRAM THE FMSTO COMPLY WITH RESTRICTION BUT DUE TO HIS INEXPERIENCE 
WITH THE ACFT FMS (2 MONTHS TOTAL ON ACFT) AND THE FACT THAT THE ACFT WAS ON A VECTOR THAT HAD TAKEN 
IT OFF THE FMS LNAV COURSE [,] THE CAPT COULD NOT PROPERLY PROGRAM THE FMS TO CAUSE THE ACFT TO LEAVE 
ALT. 

249654 THE LGT FMS HAS A NAV PROB ENRTE THAT HAS BEEN RPTED THROUGH MY COMPANY BUT NOTHING HAS 
BEEN DONE ABOUT IT. WITH THE FMS PROPERLY PROGRAMMED IT DOES NOT ALWAYS MAKE TURNS ENRTE WHICH 
COMPLY WITH THE REQUIRED ACCURACY AND TECHNIQUE TO SATISFY ATC, EVEN THOUGH THIS FMC AND ACFT ARE 
FAA CERTIFIED IT HAPPENS AT MANY DIFFERENT PLACES BUT THE LAS DIRECT GFS DIRECT HEC RTE SEEMS TO BE 
ONE OF THE WORST. 

1.1.7. Miscellaneous over-reliance on automation (5) 

303310 A DIFFERENT APCH AND LNDG RWY WAS ASSIGNED. WHILE BRIEFING THE NEW APCH WE BECAME 
DISTRACTED... WE ASSUMED THE FMS WOULD START THE DSCNT FOR US WHILE BRIEFING THE APCH. LOST 
SITUATIONAL AWARENESS MOMENTARILY AND SUBSEQUENTLY FAILED TO MEET OUR RESTR. 

198046 THE FO FLEW ACCORDING TO WHAT THE INSTS WERE TELLING HIM TO DO THINKING THE COMPUTER MUST 
KNOW MORE ABOUT THE WIND THAN WE IX). ... I THINK THE BIGGEST PROBLEM WAS MY RELIANCE ON THE FLT 
DIRECTOR V BARS WHICH CAUSED ME TO DEVIATE SO FAR FROM THE DESIRED COURSE. 

251901 UNEXPLAINED AND UNEXPECTED FMS FAILURE. ... ONE CREW MEMBER NOT MANUALLY TUNING OF ABB 
VOR AS A BACKUP. ... TOO MUCH TRUST IN THE AUTOMATION. 

330250 AS THE FMS WAS BEING PROGRAMMED FOR THE INTERCEPT BOTH PLTS NOTICED THE ACFT DSNDING 
THROUGH 10700 FT. A CLB WAS INITIATED AT 10600 FT AND THE ACFT ONCE AGAIN LEVELED OFF AT 1 1000 FT 
HUMAN PERFORMANCE CONSIDERATIONS - THE CREW WAS RECEIVING NUMEROUS UPDATED CLRNCS FOR THE 
DSCNT, THEN A TURN AT LEVEL OFF WHICH REQUIRED ADDITIONAL PROGRAMMING OF THE FMS. 

153355 THE FMS DISPLAY WAS SELECTED AND SHOWED US TO BE ON COURSE. ... APCH CTL TOLD US THAT HIS 
RADAR SHOWED US TO BE +/- 3 MI N OF COURSE. APCH CTL GAVE US A L TURN VECTOR TO RETURN TO COURSE. 
APCH CTL THEN GAVE US A R TURN VECTOR AND WE BELIEVED THAT WE WERE THEN NOT ON THE PUBLISHED ARR.IN 
FACT THE F/O REMARKED THAT WE MUST BE IN AN APCH CTL VECTOR FOR A W DOWNWIND FOR A S LNDG AT 
SEATAC. THEREFORE, WE DID NOT MAKE THE PUBLISHED TURN TO 340 DEG. 


1.2. Other automation-related problems (1) 

(Also see automation-related problems in "2. Confusion, changes, and other problems during descent/approach".) 


Incident 238398 involved misuse of automation and problems with crew coordination. 


238398 CAPT INSISTED FMS NOT BE PROGRAMMED FOR ANTICIPATED CLRNCS AND NAV AIDS NOT BE TUNED IN 
ADVANCE THIS IS TO PREVENT THE FMS FROM FLYING A RTE NOT YET CLRED FOR.’ HE SAID THE CIVET 3 PROFILE 
DSCNT WAS NOT IN THE DATABASE, a CHKED AHEAD OF TIME - IT WAS.) ... I SET THE FIRST AT OR ABOVE' OF 14000 

FT IN THE ALT WINDOW OF THE FLT CTL PANEL. THE CAPT RESET 10000 FT AND SAID SOMETHING TO ME ABOUT, 

TRUSTING THE FMSTO FLY THE PROFILE ACCURATELY.'. ..I LOOKED UP TO SEE THE ACFT PASSING THROUGH 1 2000 FT 
PRIOR TO THE AT OR ABOVE' RESTRICTION... APCH CANCELLED OUR APCH CLRNC AND ASKED US OUR SPD. ... SHORT 
OF ’OVERRIDING' THE CAPT AND TAKING PHYSICAL CTL OF THE ACFT, I'M NOT SURE I COULD HAVE DONE ANYTHING 
DIFFERENTLY... . 
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2 . Confusion, changes, and other problems during descent/approach (29) 


2.1. Last minute approach/runway change leads to significant confusion (3) 

The Cali accident involved additional workload imposed on the crew by their acceptance of a runway change during the 
automation-aided approach. This change created time pressure on the execution of procedures and led to significant crew 
confusion. 


Incident 310130 is one of the two incidents rated by all three analysts as highly relevant to the Cali accident. The other is 
368360 in section 1.1.3. See appendix 3, table 1 for the list of incidents sorted on their relevance ratings. Incident 310130 
involved not only a last minute approach/runway change, but also confusion during descent/approach, problems with crew 
decision making, confusion regarding charts, and problems with crew coordination. 


310130 AT THE LAST MIN, AFTER WE WERE VECTORED DIRECT TOWARD THE OUTER LOCATOR OC’, WE WERE CLRED 
FOR A STRAIGHT IN LNDG ON RWY 1 1 ' AND TOLD TO RPT OVER OC.' ... THE FO INITIALLY SET UP HIS RADIO ON THE 
LOC 1 10. 1, BUT THERE WAS NO LOC OR ANYTHING ON THAT FREQ. ... WE TAD BRIEFED BOTH THE ILS TO RWY 35 WITH 
A CIRCLE TO LAND AND THE LOC- VOR-DME RWY 11 APCH, BUT NOT A STRAIGHT IN APCH. THE ONLY STRAIGHT IN 
APCH WAS AN ADF LOCATOR APCH, WITH DME. ... MEANWHILE I WAS TRYING TO FIND AN APPROPRIATE APCH PAGE. 
WE SETTLED ON 1 1-2 CHART SINCE THE CILR HAD CALLED THE APCH A STRAIGHT-IN APCH.' ... I SAID 1 AM 
CONFUSED.’ I DEDNT UNDERSTAND WHY WE WERE DSNDING AND THE FO HAD ALL FLAGS WITH HIS RADIO ON THE ILS 
FREQ. ... I COULDNT FIGURE OUT WHICH APCH HE WAS USING, AND I HAD TROUBLE READING HIS CHART FROM 
ACROSS THE COCKPIT. THEN THE SO MENTIONED THAT WE HAD A 3000 FT MSA. WE WERE AT 2650 FT... THE APCH WE 
WERE FINALLY GIVEN, OR FLEW ANYWAY, DID NOT CONFORM TO ANY OF THE PLATES. ... I ACCEPTED THE CLRNC 
FOR A STRAIGHT-IN APCH, NOT KNOWING WHICH APCH. 


Incident 146645 involved not only the effect of changes late in descent/approach, but also confusion regarding charts, 
erroneous data entry, and automation turning the aircraft off course. 


146645 1 TUNED AND SET THE NA V EQUIP FOR THE ILS TO RWY 32 AND ANNOUNCED THAT THE APCH CHK WAS 
COMPLETE... HOWEVER, OUR POS AND HDG RELATIVE TO THE LOC WAS LOOKING LIKE AN EXTREMELY TIGHT TURN 
ON OR AN OVERSHOOT, SO I REQUESTED OUR CURRENT HDG AND ALT ASSIGNMENT FROM APCH. WHAT WE RECEIVED 
WAS A TURN AND CLRNC FOR THE NDB APCH RWY 30 CIRCLE TO LAND RWY 32 AT STP AND TO CONTACT ST PAUL 
TWR. WHILE HURRYING TO GET THE N A V EQUIP RETUNED AND IDENTED AND RECONFIGURING THE ACFT I WENT 
RIGHT BY THE NDB RWY 30 CHART AND TOOK THE DATA FROM THE NDB RWY 3 ST PAUL, MN, LAKE ELMO. AS WE 
WENT BY THE NDB AND STARTED TO TURN TO THE INBND HDG, SOME 70 DEG FROM OUR CURRENT HDG, WE KNEW 
SOMETHING WAS WRONG. A QUICK CHK OF THE CHART CONFIRMED THE ERROR I HAD MADE. ... THE TWR CALLED 
AND SAID APCH SHOWED US NE OF COURSE ... WE AS A FLT CREW, SHOULD NOT HAVE ACCEPTED THE CLRNC FOR 
AN APCH TO A RWY DIFFERENT FROM THE ONE WE WERE SET UP FOR, UNLESS WE WERE SURE WE HAD TIME TO SET UP 
AND CONFIGURE FOR THE APCH TO THE NEW RWY. [also see 146645 in ”1.1. 1. Automation turns aircraft off course" and 
"2.2.2. Confusion regarding charts"] 


Incident 306151 involved not only the effect of changes late in descent/approach, but also confusion, crew coordination 
problems, and loss of situational awareness. 


306151 APCH CTL ASSIGNED US THE ILS FOR RWY 7. ... THEN APCH CTL ASSIGNED US THE ILS FOR RWY 8. ... JUST 
OUTSIDE THE OM WE SWITCHED TO TWR. TWR ASKED IF WE COULD ACCFTT LNDG ON RWY 1 7R. THE CAPT AC CEPTE D 
I IMMEDIATELY ASKED WHERE RWY 1 7R WAS. THERE WAS NO RESPONSE TO MY QUESTION. ... ON THE MISSED APCH I 
HAD TO CALL 3 TIMES FOR THE CAPT TO SET MAX PWR AND FLAPS TKOF ... HE SAID HE WAS STILL THINKING ABOUT 
WHAT HAPPENED BACK THERE. I SAID, 'YOU NEED TO BE THINKING ABOUT THIS APCH NOW. FLY THIS APCH NOW.' ... 
THE FACT IS, NEITHER OF US KNEW WHERE RWY 17R WAS IN RELATION ”T> RWY 8. ... THE CAPT ACCEPTED A CLRNC 
WHEN HE SHOULD NOT HAVE. ... I DID NOT TAKE THE INITIATIVE TO STOP A SIT I WAS UNCOMFORTABLE WITH. 

2.2. Other confusion during descent/approach (11) 

2.2.1. Name confusion (5) 


(Also see section "1.1.4. Name confusion using automation" in section ”1.1. Over-reliance on automation.”) 

The Cali accident involved two distinct instances of name confusion. First, the arrival was named ROZO 1, instead of TULUA 1, 
which appeared to confuse the captain. According to the NTSB report (NTSB, 1996), ”CVR (cockpit voice recorder) evidence 
reveals that the crew may have expected the standard STAR naming convention to be used with respect to the ROZO 1 Arrival and 
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may have incorrectly believed that ROZO was located at the beginning of the route.” The second instance of name confusion 
involved crew confusion of the identifier R for ROZO, which the automated system interpreted to mean ROMEO. 

Among the following incidents involving name confusion, several also involved miscommunication between the crew and 
controller. 


In addition to name confusion, incidents 310228 and 274820 involved problems with situational awareness. 


310228 UNBEKNOWNST TO US THERE ARE 2 APCHS, ONE NAMED THE (GPS) VOR/DME RWY 33L AND THE OTHER 
NAMED VOR DME OR GPS A. WE UNDERSTOOD THE CLRNC TO BE FOR THE (GPS) VOR DME RWY 33L WHICH HAS AN 
INBOUND COURSE OF 342 DEGS, WHILE THE APCH CTLR UNDERSTOOD THE CLRNC TO BE FOR THE VOR DME OR GPS A 
APCH WHICH HAS AN INBOUND COURSE OF 310 DEGS. SINCE WE WERE LOOKING TO INTERCEPT THE 342 DEG COURSE 
WE OVERSHOT THE 3 10 DEG COURSE. TWR TOLD US WE WERE L OF COURSE. ... ONE FURTHER POINT OF CONFUSION: 
THE FINAL APCH FIX FOR THE VOR DME OR GPS-A APCH IS BEEJE AND THE FAF FOR THE (GPS) VOR DME RWY 33L IS 
MEACH. THESE SOUND VERY SIMILAR AND COMBINED WITH THE ALMOST IDENTICAL APCH NAMES CREATES A 
STRONG POTENTIAL FOR CONFUSION. 

274820 I ANTICIPATED VFR CONDITIONS...I WAS NOT AS MENTALLY PREPARED FOR THE UPCOMING INST APCH AS I 
WOULD HAVE BEEN OTHERWISE. WE DECIDED THE VOR DME C APCH WOULD BE BEST... HOWEVER, AFTER 
INTERCEPTING THE FINAL APCH COURSE MCGRATH RADIO BEGAN TO QUESTION OUR LOCATION AND WE SOON 
LEARNED THAT WE HAD BEEN CLRED FOR THE VOR DME 16 APCH INSTEAD OF THE VOR DME C APCH. 

142553 THE WORD "VIS" IS INAPPROPRIATE IN THE NAME OF IFR/VFR APCH. BOTH PLTS HEARD "VIS APCH" NOT 
"QUIET BRIDGE VIS APCH!". PLTS ARE TRAINED TO KEY ON THE PHRASE "CLRED FOR VIS APCH" AND CAN BE MISLED 
IF SUCH WORDING IS CONTAINED IN AN APCH WHICH IS NOT A PURE VIS. 

310989 I HAD ARRIVED AT ARCFH WITHOUT ANY ADDITIONAL RTE CLRNC. IT IS NOT CUR THAT THE ORIGINAL CLRNC 
TO INTERCEPT THE SFO 095 DEG STILL APPLIES AFTER BEING AMENDED TO FLY DIRECT ARCHI. ... I QUERIED 134.5 
ABOUT OUR CLRNC AFTER ARCHI. BAY APCH RESPONDED, INTERCEPT THE FINAL APCH COURSE.' THIS WAS THE 
FIRST TIME I HAD HEARD THE TERM FINAL APCH COURSE' REFERRING TO THE SFO 095 DEG RADIAL. I INQUIRED 
AGAIN FOR CLARIFICATION , THAT IS NOT CLR TO ME, SHOULD WE INTERCEPT THE SAN FRANCISCO 095 DEG RADIAL?' 

THE CTLR INSISTED ON USING ONLY THE TERMINOLOGY FINAL APCH COURSE' AND WOULD NOT RESPOND 
OTHERWISE TO MY REPEATED ATTEMPTS TO RECONCILE MY UNCERTAINTY... 

335430 WE THOUGHT AT THIS TIME THAT THE VOR 17 APCH WAS IN USE. MANCHESTER HAS 2 VOR APCHS TO RWY 17 
(VOR 17, CHART 13-2) WHICH U TILIZES THE MHT VOR AND THE VOR DME 17 WHICH USES THE CONCORD VOR (APCH 
CHART 13-3). ... AS WE INTERCEPTED THE FINAL APCH HE ADVISED US THAT WE WERE 1/4 MI R OF COURSE AND THAT 
THE APCH USED THE CONCORD VOR. 


2.2.2. Confusion regarding charts (4) 


Incident 352618 not only involved confusion regarding charts, but also involved confusion regarding the detailed 
representation of terrain on charts. It also involved complacency in the presence of terrain. See the excerpt in section 3.1.1. 

352618 WE FELT THAT THERE WAS SOME AMBIGUITY ABOUT TERRAIN CONTOURS DEPICTED ON THAT CHART AS 
COMPARED TO THE AREA CHART FOR LAS VEGAS. THE APCH CHART SHOWS A CONTOUR INTERVAL MARKED 3000 FT, 
AND THE SAME INTERVAL IS MARKED 4000 FT ON THE AREA CHART. ...I SAW THE NUMBER 3000 FT AND FORGOT 
THAT IT WAS A CONTOUR INTERVAL EXTENDING FROM 3000 FT TO 4000 FT IN THE AREA WE WERE IN. [also see 3526 1 8 
in "3.1.1. Complacency, loss of situational awareness, slow reaction"] 

Narrative 226114 is one of those rare ASRS reports that describes a situation that seems unsafe to the reporter, as opposed to a 
particular incident. This situation, in Juneau, Alaska, is of such importance that special procedures are being developed for it 
(Steenblik, 1998). 

226114 THIS IS THE 3RD MAJOR ACCIDENT ON THE APCH FOR JNU (LDA 1 RWY 8). ALL 3 ACFT (ACR-LGT, ACR-LTT, 
MIL-MLT) HIT WITHIN 1/2 MI OF EACH SEPARATE ACCIDENT SITE. ALL 3 CREWS APPARENTLY MISIDENTED B ARLO, 
THE FINAL APCH FIX. I KNOW FROM MY PERSONAL EXPERIENCE, THAT THE APCH PLATE TAKES A LOT OF STUDYING 
BECAUSE OF All. THE POSSIBLE TRANSITIONS. ... I THINK HAVING LESS CLUTTER ON THE PLATE WOULD BE A BIG 
HELP. ...CHART IS CONFUSING. 
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In addition to confusion regarding charts, incident 146645 also involved automation turning the aircraft off course (see further 
excerpt in section 1.1.1.) and a last minute approach/runway change leading to significant confusion (see further excerpt in 
section 2.1.). 


146645 WHILE HURRYING TO GET THE N A V EQUIP RETUNED AND CDENTED AND RECONFIGURING THE ACFT I WENT 
RIGHT BY THE NDB RWY 30 CHART AND TOOK THE DATA FROM THE NDB RWY 3 ST PAUL, MN, LAKE ELMO. AS WE 
WENT BY THE NDB AND STARTED TO TURN TO THE INBND HDG, SOME 70 DEG FROM OUR CURRENT HDG, WE KNEW 
SOMETHING WAS WRONG, [also see 146645 in "1.1.1. Automation turns aircraft off course” and "2.1. Last minute 
approach/runway change leads to significant confusion"] 


365456 THE EASTSIDE ONE ARR IS THE ONLY FMS ARR KNOWN TO THIS AVIATOR TO RETAIN DSCNT CLRNC AFTER 
BEING CLRED FOR FMS APCH. CERTAINLY, A SIT FOR CONFUSION WHEN COMPARED TO MOST, IF NOT ALL OTHER FMS 
APCHS I M FAMILIAR WITH. ... UPON REVIEWING THE FMS STAR CHART, HE SAW THE NOTE ABOUT MAINTAINING THE 
LAST ASSIGNED ALT UNTIL PASSING KAYOH, HOWEVER, HE STILL THINKS THAT THE STAR IS MISLEADING AND ITS 
FORMAT CONFUSING. 


2.2.3. Confusion due to use of wrong data (2) 


335098 I WAS SURPRISED TO FIND MY DME INDICATING 13.5, OR 1.5 MI PAST FERNS. I IMMEDIATELY INITIATED 
DSCNT TOWARD 1 200 FT, OUR NEXT STEP-DOWN ALT. SHORTLY, THE FO INDICATED SOMETHING WAS NOT RIGHT. .. . I 
THEN DISCOVERED MY DME HOLD FEATURE WAS ENGAGED AND IMMEDIATELY SELECTED IT OFF. MY DME THEN 
READ APPROX 10.5 AND I REALIZED WE HAD NOT YET REACHED FERNS. ... WE HAD DSNDED SEVERAL HUNDRED FT 
BELOW THE MINIMUM ALT OF 2000 FT. 


Incident 212324 involved data entry error and use of the wrong navigational beacon. 


212324 I HAD ABE IN VOR AND WE WERE SUBSEQUENTLY CLRED THE LOC BACK COURSE APCH TO RWY 24. 1 FAILED 
TO TUNE IN ETX VOR FOR DME INFO. CAPT WAS FLYING AND NOTICED DME WAS ABOUT 10 DME. HE BEGAN DSCNT 
DOWN TO THE MD A OF 760 FT MSL. AT ABOUT 1000 FT MSL (600 FT AGL)TWR CALLED AND SAID WE WERE WELL 
BELOW THE PROPER ALT FOR THE APCH AND WE SHOULD CLB BACK IMMEDIATELY. 


2.3. Forgot speed brakes (2) 


The Cali accident involved execution of a terrain escape maneuver without retraction of the speed brakes. Two of the incidents 
relevant to the Cali accident involved forgetting to retract the speed brakes. 


334866 I AM NEW ON THE B757...WE HAD TO USE SPD BRAKES BTWN ARCHI AND GAROW INTXNS TO GET DOWN TO 
PROFILE. ...AIRPLANE DIDNT STOP AT THE PROFILE, IT KEPT DSNDING AND BUSTED THE 4000 FT RESTR AT GAROW 
INTXN BY 400 FT. IT TOOK ME A FEW SECONDS TO REALIZE WHY - AS THE ACFT APCHED THE PROFILE, I HAD NOT 
RETRACTED THE SPD BRAKES SINCE I WAS DISTRACTED AND THERE ARE NO INDICATIONS OF SPD BRAKE 
DEPLOYMENT TO REMIND YOU. 


Incident 280233 not only involved forgetting the speed brakes, but also involved a GPWS warning. 


280233 I CALLED FOR GEAR-DOWN 1 AND EXTENDED SPD BRAKES TO AID DSCNT. ... AT 8200 FT... I RAISED THE NOSE 
TO ARREST SINK RATE AND DECELERATE AT 7200 FT. ... I APPLIED THRUST AND NOTICED I WAS USING 
CONSIDERABLY MORE THAN NORMAL TO MAINTAIN LEVEL FLT. ... AT APPROX 200 KTS, I GOT THE STALL SHAKER... I 
INSTINCTIVELY LOWERED THE NOSE AND ADDED THRUST. AT 6500 FT THE GPWS ISSUED A TERRAIN’ WARNING. THE 
FE THEN ALARMED ME THE SPD BRAKES WERE STILL EXTENDED. 

2.4. Other problems with changes late in descent/approach (4f 


Incident 275413 involved not only the effect of changes late in descent/approac a, but also loss of data when other data are 
entered into the automated system. 


275413 AT THIS POINT, TO COMPLY WITH THE 210 KIAS SPD REQUEST BY ATC, THE FO SELECTED THE VERT SPD 
FUNCTION ON THE DIGITAL FLT GUIDANCE PANEL, SETTING 600 FPM DSCNT RATE AND CHANGING THE ’SPD SELECT 
WINDOW TO SHOW 210 KTS. BECAUSE OF A FLAW IN THE MD-88’S DIGITAw FLT GUIDANCE PROGRAM, LAST SECOND 
CHANGES IN ANY OF THE VERT CTL FUNCTIONS WILL REMOVE THE ALT LEVEL OFF COMMAND PREVIOUSLY 
SELECTED. THE ALT WARNING SIGNAL AND A TCASD TA’ GOING OFF SIMULTANEOUSLY AT 7700 MSL BROUGHT OUR 
ATTN TO THE DEV. [also see 275413 in "1.1.2. Loss of data when other data ate entered”] 
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310373 WE WERE BEING VECTORED FOR THE 'CANARSIE APCH' (VOR RWY 13L/R) AT J FK. ...T HE CRI VOR WAS AT 12 
O’CLOCK (STRAIGHT AHEAD), ABOUT 5 MI. THE PNF INDICATED TO ME SHORTLY THEREAFTER THAT THE USUAL 
VISUAL REF POINTS JUST BELOW US WERE NOT WHAT THEY SHOULD BE I CHKED MY RMI NEEDLES (WE BOTH HAD 
THE CRI VOR TUNED AND IDENTED) AND SAW THEM SWINGING OFF TO THE R, ABOUT 2.5 DME. I RECHKED MY WAV' 
(FMS) DISPLAY, AND IT WAS NO LONGER AT 1 2 O'CLOCK, BUT NOW WAS OFF TO THE R (SEE NOTE). IT HAD BEEN AT 1 2 
O'CLOCK, 5 MI, ONLY 4-5 SECONDS PRIOR TO THAT. WE INITIATED A R TURN AND I IMMEDIATELY SWITCHED TO "RAW 
DATA' TO COMPLETE THE APCH. ... WHEN VECTORS ARE CHANGED CLOSE TO THE FAF, MORE THAN THE NORMAL 
XCHK MUST BE ACCOMPLISHED BY THE PLT WHO HAS TO QUICKLY GO FROM FMS INFO TO RAW DATA INFO, 
DECIPHER, SWITCH DISPLAYS, FLAG THE APCH, CONFIGURE, DSND TO MDA, ETC. 

2 1 7430 RWY 36R ILS WAS BRIEFED. AT GILMORE, WE WERE CLRED DIRECT TO AULON FOR THE ILS RWY 9. 1 TOLD 
THE FO TO BRIEF ME ON THE APCH WHILE I ACTIVATED THE SECONDARY FLTPLAN IN THE FMS. WE WERE CLRED TO 
DSND TO 3000 AND I BEGAN A RAPID DSCNT TO THAT ALT AS THE ARPT WAS NOW VERY CLOSE. ... I GLANCED 
QUICKLY AT THE APCH PLATE FOR ELS RWY 9 AND SAW 1 500 AS THE INTERMEDIATE APCH ALT. ... WE WERE ASKED 
OUR ALT BY APCH AND WE RPTED 1 500 FT. THEY ADVISED THEY WERE RECEIVING A LOW ALT ALERT. 

335282 WHILE TURNING ONTO THE FINAL APCH PATH, WHILE EM CONTINUOUS MODERATE TURB AND MODERATE 
RAIN SHOWERS, THE APCH CTLR INFORMED US THAT A PRECEDING FLT MISSED THE APCH AND INFORMED US THAT 
WE MIGHT HAVE BETTER LUCK WITH THE ROSSLYN LDA RWY 1 8. THE CTLR THEN CURED US FOR THE ROSSLYN LDA 
RWY 1 8 APCH. SINCE THIS WAS THE FIRST WE HEARD THAT THE ROSSLYN LDA WAS IN USE, WE WERE VERY RUSHED 
PREPARING FOR THE APCH. ... [Pan of the problem is] THE LACK OF STORED APCH INFO FOR THE APCHS TO RWY 1 8. ... 
THE FLC EITHER HAS TO FLY THE APCH USING THE APCH PLATE ONLY OR TAKE THE TIME (WHI CH WA S NOT 
AVAILABLE) AND ENTER EACH WAYPOINT INTO THE FMC. THE APCH PLATE ITSELF MUST BE BRIEFED AND THESE LAST 
MIN CHANGES CREATE A VERY HIGH WORKLOAD, HIGH STRESS, ENVIRONMENT IN THE COCKPIT. 

2.5. Miscellaneous problems during descent/approach (9) 

282707 HAD DSCNT CONTINUED THERE WOULD HAVE BEEN A COLLISION. ... ONE BECOMES CONFIDENT AND 
TRUSTING WHEN BEING CILED INTO AN ARPT OF THE SIZE OF SFO. WE RELY ON TFC SEPARATION. IT IS ALSO A TIME 
THAT IS VERY BUSY FOR FLC - SETTING UP FMS, FINAL APCH BRIEFINGS, CHKLISTS, MONITORING APCH FREQS TO 
HEAR WHAT OTHER ACFT ARE DOING. THANKS TO A DILIGENT AND OBSERVANT FO AND TCASD, THIS POTENTIAL 
COLLISION WAS AVOIDED. 

279493 FLYING UP HUDSON RIVER ... WE WERE THEN CLRED TO FLY THIS APCH. OUR APCH PLATE SPECIFIES 
REMAIN ABOVE 2000 FT AS LONG AS POSSIBLE' BUT NO OTHER ALT RESTRS. ... CTR TOLD US WE WERE NOT CLRED 
FOR LOWER AND TO CLB BACK TO 3500 FT. ... QUALITY ASSURANCE SPECIALIST STATED THE RPTR IS CORRECT. HE 
CAN DSND TO 2000 FT. 

309352 THE APCH PLATE SPECIFIES FOR US TO REMAIN 'AT OR ABOVE 2000 FT AS LONG AS POSSIBLE.' ... APPROX 
ABEAM THE WORLD TRADE CTR, THE APCH CTLR . .. ASKED WHAT OUR ALT WAS AND I REPLIED WE WERE PASSING 
2500 FT GOING DOWN TO 2000 FT. HE RESPONDED THAT WE WERE SUPPOSED TO MAINTAIN 3000 FT MSL UNTIL 
ABEAM CENTRAL PARK. 

363380 ACFT WAS AT 2800 FT DSNDING. ... REMEMBERED ALT ASSIGNED AND A CLB STARTED BACK TO 3000 FT. 

IN FUTURE, RECOMMEND THE PF USE AUTOPLT TELL ESTABLISHED ON FINAL APCH COURSE IN ORDER TO FREE UP 
MENTAL WORKLOAD TO BETTER IMPROVE SITUATIONAL AWARENESS. 

99108 WE WERE ADVISED BY COAST APCH THAT THE RWY LIGHTS ON ALL RWY S WERE NOT TURNING ON. HE 
THEN ASKED APCH WHAT TIME SUNSET WAS, AND THEY INFORMED HIM 3 MINS AGO. ... WE COULD STILL SEE THE 
ARPT, AND IT WAS STILL LIGHT OUT, SO THE CAPT TOLD APCH WE WOULD CONTINUE 

242545 I WAS TOLD THAT OUR CLRNC WAS TO THE HUSON FIX, VIA THE 13 DME ARC. MAINTAIN 13000 FT, EXPECT 
NO DELAYS FOR THE VOR DMEA APCH. THIS IS THE CLRNC THAT MY FO RECEIVED WHILE I WAS OFF THE AIR. HE 
STATES THAT HE MISTOOK THIS FOR AN APCH CLRNC. 

84811 I EXPLAINED ABOUT BEING ON A VISUAL APCH AND HE SAID I SHOULD HAVE RECEIVED THE XING 
RESTRICTION. I CONFERRED WITH MY FIRST OFFICER AND WE BOTH AGREED THAT WE HAD NOT RECEIVED THE 
RESTRICTION. 

1 15883 MOUNT VERNON VISUAL APCH TO RWY 36 WASHINGTON NATL. WASHINGTON APCH FAILED TO CHANGE FLT 
OVER TO TWR. FLT CREW DID NOT REALIZE THAT RADIO WAS TUNED TO APCH AND FAILED TO REQUEST LNDG CLRNC 
FROM TWR. 
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307161 WHILE BEING VECTORED FOR AN NDB APCH AND WHILE ON DOWNWIND, THE CREW WAS TOLD BY A7C, 
UNITED STATES CUSTOMS REQUIRES NXXXXX TO BE DIVERTED TO FT PIERCE SO AS TO CLR CUSTOMS AND THAT 
LNDG CLRNC TO MELBOURNE IS DENIED. ' ATC WAS ADVISED BY THE CREW THAT THE AIRPLANE HAD NOT LANDED IN 
A FOREIGN COUNTRY AND THAT 4 FAA INSPECTORS WERE ONBOARD THE ACFT. 

Tcnam. aifli i iaa ££— 1 12 } 

3.1. GPWS alarms (15) 


The many GPWS alarms in the ASRS database illustrate the kinds of experiences flight crews have with this terrain 
avoidance system. Previous experience with false GPWS alarms can be a factor in accidents involving controlled flight 
into terrain (Majikas, 1995). 

3.1.1. Complacency, loss of situational awareness, slow reaction (5) 


351 150 APPROX 30 MI S OF LAS, RECEIVED CLRNC FOR VISUAL APCH RWY 1R, VASI OTS. DISCUSSED WITH FO 
IMPLICATIONS OF DSNDING VISUALLY IN MOUNTAINOUS TERRAIN OFF PUBLISHED RTE, PLUS NO GS INFO. WE WERE 
LULLED INTO A FALSE SENSE OF SECURITY BY FAMILIARITY OF AREA, GOOD VISIBILITY, AND TERRAIN BEING WELL 
LIT BY FULL MOON. APPROX 10 MI S OF LAS ON THE ISO DEG RADIAL, CTU* ISSUED A BELOW MVA’ ALERT WE WERE 
AT 4100 FT MSL. SECONDS LATER, A GPWS WARNING TERRAIN, TERRAIN ONLY SOUNDED WITH TERRAIN IN SIGHT. 
AN IMMEDIATE CLB WAS INITIATED... 


Incident 352618 also involved confusion regarding charts. See the excerpt in section 2.2.2. 


352618 THE CAPT STATED THAT WE WERE GOOD TO DSND NOW TO 4 1 00 FT. I COMMENTED THAT THE AIRSPACE THAT 
PROTECTED US AT 4100 FT WAS ONLY VALID ONCE WE WERE ESTABLISHED ON FINAL AND OVER THE FIX INBOUND. HE 
SAID HE WAS SURE IT WAS SAFE.. THE CAPT SEEMED VERY CONFIDENT AND NOTHING IN HIS MANNER SIGNALED 
THAT I SHOULD BE AT ALL CONCERNED ABOUT HIS JUDGEMENT. I REMEMBERED THINKING EARLIER THAT HE SEEMED 
LIKE A REALLY GREAT GUY TO FLY WITH: VERY PROFESSIONAL AND SELF-ASSURED, WITH VERY GOOD PEOPLE 
SKILLS TOO. ... ALMOST IMMEDIATELY THE CAPT SAID SOMETHING ABOUT A MOUNTAIN BEING VISIBLE OUTSIDE THE 
WINDOW. I LOOKED OUT AND OUR LNDG LIGHTS WERE CLEARLY ILLUMINATING A LARGE PEAK BELOW OUR NOSE. 
THE CAPT SAID, THE RADIO ALT IS SHOWING 1000 FT, LETS GET OUT OF HERE! 1 1 DISCONNECTED THE AUTOPLT AND 
INITIATED A CLB AT TOGA THRUST BACK UP TO 6000 FT, SHORTLY AFTER I STARTED THE CLB, THE GPWS CALLED 
TERRAIN, TERRAIN!’ [also see 352618 in "2.2.2. Confusion regarding charts*] 

363536 CTLR GAVE US A VECTOR... SHORTLY AFTER THIS THE GPWS ALERTED US TO TERRAIN, TERRAIN.' AN 
IMMEDIATE CLB WAS INITIATED AND THE ACFT WAS SUBSEQUENTLY LANDED SAFELY. ... THIS ACR DOES NOT USE 
ANY TERRAIN AWARENESS IN THEIR APCH BRIEFINGS. ... FO ADMITS TO IT BEING VERY MUCH OF A RUSHED 
ATMOSPHERE AND ADMITS TO A LOSS OF SITUATIONAL AWARENESS IN EVENT. 


Incident 82787 also involved lack of detailed terrain information on approach rharts. 

82787 I YELLED, "CLB NOW," VERY FORCEFULLY. WHEN THERE WAS NO IMMEDIATE RESPONSE, I HIT AUTOPLT 
DISCONNECT AND PULLED UP .... THIS HAS HAPPENED BEFORE IN THIS LOCATION. A FOREIGN CARRIER HAS 
CONDUCTED EXTENSIVE RESEARCH. THERE IS A 5637’ HILL IN THE AREA 'HAT SHOWS ON THE DEP CHART, BUT IT IS 
NOT DEPICTED ON THE APCH CHART. THE CHART PUBLISHER HAS BEEN CONTACTED AND THEY SAID THEY WOULD 
INCLUDE THIS HILL ON FURTHER APCH CHARTS. RPTR WAS PARTICULARLY DISTURBED BY THE FACT THAT HIS 
SKILLED AND COMPETENT F/O FROZE IN THE MACHINE MODE AND TRIED TO COMPLY WITH THE GPWS INSTRUCTIONS 
WITH THE AUTOPLT CTLS RATHER THAN EXECUTING AN IMMEDIATE PULLUP MANUALLY. 

346137 AFTER LEVELING AT 4600 FT MSL FOR 15-20 SECONDS THE GPWS (MODE 2) TERRAIN, TERRAIN' ALERTED, 
BOTH PLTS WERE SURPRISED AS WE HAD ASSUMED THAT 4600 FT WAS MINIMUM VECTOR FOR THAT AREA. AS GPWS 
ALERTED SECOND TIME, PLTS INITIATED ESCAPE MANEUVER WITH GAR 'TRUST FOR MAX CLB ANGLE 
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3.1.2. Miscellaneous GPWS alarms (10) 


325026 PASSING THROUGH 3000 FT MSL, WE RECEIVED A GPWS ALERT WARNING OF TERRAIN, TERRAIN.' I 
IMMEDIATELY INITIATED THE PRESCRIBED PROC OF FIREWALL PWR AND PITCH TO 20 DECS NOSE UP. THE WARNING 
CONTINUED FOR ABOUT 10-15 SECONDS. ... THE ACFT HAD TO HAVE THE ENGS INSPECTED AS THE PF HAD 
FIREWALLED' THE ENGS IN THE CLB MANEUVER, OVERTEMPING THEM IN THE PROCESS THE ACFT WAS FERRIED 
BACK TO ATL THE NEXT DAY. ... THE ACFT HAD ITS SPOILER-SPD BRAKES DEPLOYED. THE CREW, IN COGNIZANCE OF 
THE LATEST ACCIDENT IN S AMERICA, DID NOT FAIL TO REMEMBER THE ACFTS CONFIGN AND RETRACTED THE 
SPOILERS... THE AREA SE OF HUNTSVILLE ARPT HAD A HISTORY OF ERRONEOUS GPWS WARNINGS. 

174048 AFTER DSNDENG TO 3800' MSL THE GPWS ISSUED A CONTINUOUS AURAL AND VIS "TERRAIN" ALERT. SINCE 
WE COULD NOT IMMEDIATELY DETERMINE VISUALLY THAT WE HAD ADEQUATE TERRAIN CLRNC AND THE RADAR 
ALTIMETER INDICATED DECREASING GND CLRNC, WE INITIATED A CLB ... ROA HAS BEEN DESIGNATED BY THE FAA AS 
A SPECIAL ARPT DUE TO MOUNTAINOUS TERRAIN AND HIGH OBSTRUCTIONS IN THE AREA. BECAUSE OF THE SPECIAL 
NATURE OF THIS ARPT, COMPANY HAS 9 PAGES OF INSTRUCTIONS TO AID FLT CREWS IN ARR, APCH AND DEP PROCS. 

201005 WE WERE INSTRUCTED TO CLB TO 8000 FT, AND TURN TO A HDG OF 180. AFTER WE WERE LEVEL AT 8000 FT, 
AND ON THE 180 HDG FOR SOME TIME, WE RECEIVED A CONTINUOUS TERRAIN WARNING FROM THE GPWS. THIS WAS 
OVER A COMPLETELY DARK AREA ... I INITIATED A CLB AND THE FO ADVISED APCH CTL ... THE APCH CTLR STATED 
THAT HE WAS PROVIDING TERRAIN CLRNC FOR US AND FOR US TO MAINTAIN 8000 FT... BUT DID GIVE US A TURN 
TOWARD THE CTR OF THE VALLEY. WE TURNED, BUT IGNORED THE ALT INSTRUCTION AND CONTINUED THE CLB 
ENOUGH TO SILENCE THE GPWS WARNING. A CILR S EX POST FACTO STATEMENT THAT HE IS PROVIDING OBSTACLE 
CLRNC IS SMALL COMFORT WHEN WHOOP, WHOOP, PULL UF IS RINGING IN ONE’S EAR. 

317197 OVER THE RIDGE OF HILLS WE WERE STARTLED TO HEAR OUR GPWS ANNOUNCE TERRAIN, TERRAIN THIS 
STARTLED US SINCE AS I MENTIONED EARLIER WE WERE AT AN ATC CLRED ALT AND HDG AND COULD VISUALLY SEE 
THAT WE WERE IN NO DANGER OF BEING NEAR TERRAIN. OF COURSE THE CAFT REACTED TO THE WARNING 
IMMEDIATELY... 

362229 INBOUND TO LAS. ..WE RECEIVED A GPWS TERRAIN WARNING. IN COMPLIANCE WITH OUR COMPANY PROCS. 

I INSTRUCTED THE FO TO INCREASE PWR AND CLB. ...SINCE WE WERE AT THE MV A OF 6100 FT MSL, I FEEL THAT WE 
WERE GIVEN AN INCORRECT TERRAIN WARNING. 

300252 AS THE CTLR GAVE US THE HDG TO INTERCEPT THE LOC COURSE HE ADVISED THAT THE HDG AND ALT MIGHT 
CAUSE A GPWS WARNING THAT HE CHARACTERIZED AS SPURIOUS, BUT THAT WE WOULD CLR THE TERRAIN. ... I HAD 
BRIEFED THAT WE WOULD RESPECT ALL GPWS WARNINGS DURING OUR OPS AT RNO. 

329185 DURING APCH PHASE OF ARR INTO RNO WE EXPERIENCED A GPWS WARNING OF TERRAIN FOLLOWED BY 
PULL UP.' ... THE CAFT WAS TOLD BY A SUPVR THAT THIS HAPPENS ALL THE TIME OVER HAZEN MOUNTAIN. THE 
MOUNTAIN IS ABOUT 7800 FT HIGH WITH A TWR ON IT SO THE GPWS GOES OFF. 

354277 WHILE ON ARR INTO SLC, WE WERE CLRED TO DSCNT TO 9000 FT. AS WE DSNDED THROUGH 9500 FT IN IMC 
OUR GPWS TERRAIN WARNING BEGAN SOUNDING CONTINUOUSLY SO WE INITIATED A CLB AND INFORMED APCH WE 
WERE CLBING TO 1 1000 FT. THE CTLR RESPONDED THAT 9000 FT WAS ABOVE MV A AND STARTED TO GIVE US AN 8000 
FT XING RESTR FOR OUR INST APCH TO RWY 34L. ... I DONT THINK THAT THIS PARTICULAR CTLR FULLY 
APPRECIATES THE EFFECT THAT THE TERRAIN WARNING HAS ON A FLC ESPECIALLY AT NIGHT, UNDER INST 
METEOROLOGICAL CONDITION, AND IN MOUNTAINOUS TERRAIN. 

355364 WE RECEIVED A GPWS WARNING TOO LOW, TERRAIN.' FLC IMMEDIATELY PERFORMED ESCAPE MANEUVER 
... WE BELIEVE THAT EVEN THOUGH WE WERE ABOVE MV A, THE COMBINATION OF RISING TERRAIN AND ACFT DSCNT 
RATE MAY HAVE CAUSED THE GPWS TO GIVE ITS WARNING. 

242560 JUST AS WE CROSSED OVER THE PEAK OF THE MOUNTAINS BELOW US WE RECEIVED A GPWS TERRAIN' 
WARNING. ... THE UNDERLYING TERRAIN WAS CLRLY IN SIGHT AT ALL TIMES AND AN EVASIVE CLB MANEUVER WAS 
NOT NECESSARY. I HAVE RECEIVED MANY UNNECESSARY GPWS WARNINGS IN THE TRI AREA, ALL OF WHICH HAVE 
OCCURRED WHILE UNDER RADAR VECTORS FOR A VISUAL OR INST APCH. THE MINIMUM VECTORING ALTS IN THE 
AREA ARE OBVIOUSLY TOO LOW AND SHOULD BE RAISED IMMEDIATELY. I DO NOT CONSIDER THESE TO BE FALSE 
GPWS TERRAIN WARNINGS BECAUSE THE SYS APPEARS TO BE FUNCTIONING AS INTENDED. THE ATC VECTORING ALTS 
SIMPLY BRING THE ACFT TOO CLOSE TO RAPIDLY CHANGING TERRAIN. EVERY TIME A PLT RECEIVES AN 
UNNECESSARY GPWS WARNING HIS CONFIDENCE IN THE SYS IS UNDERMINED, MAKING HIM LESS LIKELY TO 
RESPOND PROMPTLY AND PROPERLY IN THE CASE OF A VALID WARNING. WE WELL CONTINUE TO EXPERIENCE CFIT 
ACCIDENTS UNTIL UNNECESSARY GPWS WARNINGS ARE ELIMINATED... 
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3.2. Other terrain-related incidents (4) 


Incident 156284 involved confusion regarding charts in context of terrain. 


156284 WHILE CLBING THROUGH 12500’ MSO APCH ASKED US TO CROSS 8 NE ON V187 AT 1 1000'. I RESPONDED THAT 
IT WOULD BE NO PROB, WE WERE ALREADY ABOVE 1 1 000’. THE CILR STATED THAT WE WERE TO MAINTAIN 1 1 000 
TILL THE MSO AND NOW TO MAINTAIN 1 1000’ TILL 8 NE OF MSO VOR ON VI 87. BOTH THE F/O AND MYSELF WERE 
SURPRISED BECAUSE OF THE CHARTS AND TERRAIN IMPLIED THAT IT WAS AN AT/OR ABOVE CLRNC. 


Incident 297695 involved problems with crew coordination. 


297695 I MISREAD THE DEP TO MAKE A L TURN AT 9000 FT WHEN I SHOULD HAVE MADE A R TURN. WAS CLR WX AND 
IN A MOUNTAINOUS AREA WHERE I WAS LOOKING OUTSIDE TRYING TO AVOID ANY TERRAIN OR OTHER ACFT AND 
SIMPLY MADE THE WRONG TURN. ... I BELIEVE IT COULD HAVE BEEN PREVENTED BY BETTER CREW COORD PRIOR TO 
DEP... BOTH PLTS WERE THE CAPTS .. . THE BRIEFING BY THE CAPT FLYING MAY HAVE BEEN MORE THOROUGH IF HE 
WERE FLYING WITH A COPLT. 


Incident 184446 involved procedures not followed and lack of detailed obstacle information on approach charts. 


1 84446 THE FLC DID NOT REVIEW OR HAVE THE SAT RADAR- 1 APCH PLATE DISPLAYED. ... [The controller] SAID TO 
DSNDTOOUR MDA OF 1200 FT, AND KEPT TELLING US TO CORRECT HDGS TO THE R AS WE WERE L OF CENTERLINE ... 
WE STARTED PICKING UP THE GND AND THEN SAW A VERY TALL TWR' AT 1459 FT AT OUR 2 O’CLOCK POS WHICH WAS 
APPROX 1/2- 1 MI AWAY (THE TOP OF THE TWR BEING MUCH HIGHER THAN OUR ALT). JUST THEN THE CILR GAVE US 
ANOTHER HDG CORRECTION TO THE R TO CENTERLINE WE TOLD HER THAT WE COULD NOT ACCEPT THIS HDG 
BECAUSE IT WOULD TAKE US INTO A TWR. SHE SAID CHK YOUR MDA ALT OF 1200 FT. SHE APPARENTLY DID NOT 
KNOW ABOUT THE 1459 FT TWR. ... I CALLED SAT APCH ON ARR SAT AND SPOKE WITH A SUPVR. HE APPARENTLY WAS 
NOT AWARE OF A PROBLEM NOR OF THE RADIO TWR ON THE RWY 21 APCH. 


Incident 264952 involved terrain-related data. 


264952 I BECAME EXTREMELY ALARMED WHEN 1 LEARNED THAT HIS CALCULATIONS WERE BASED ON THE USE OF 
MINIMUM SAFE ALT DATA AS DEPICTED ON COMMERCIAL APCH PLATES ,\ND NOT ON TOPOGRAPHICAL CHARTS OR 
COMPLETELY ON MEA INFO FROM ENRTE CHARTS. ... THE COMPANY RECENTLY FIRED 2 CREWMEMBERS FOR 
REFUSING TO TAKE A DRIFTOOWN DEPENDENT FLT THROUGH THE ROCKIES. THEY ASSERT THEY WERE NOT GIVEN 
ENOUGH TIME OR INFO TO COMPUTE THE ACFT PERFORMANCE FOR THE TRIP, AND HENCE REFUSED IT. THIS IS THE 
CLIMATE OF THE PLACE WHERE I WORK: EXTREME DURESS AND CONSTA NT FEAR OF LIVELIHOOD LOSS. . . . I DO NOT 
BELIEVE THAT THIS DATA HAS BEEN EVEN SUBMITTED TO THE FAA, LET AJjONE SCRUTINIZED BY AN ENGINEERING 
TEAM. 

4. Problems with operations in foreign airspace (6) 

4.1. Problems with operations in Latin America (4) 


Incidents 310143 and 140711 involved operations in the vicinity of Cali, Colombia, near the site of the crash of Flight 965. 

310143 ATC CLRED US DIRECT TO THE CALI VOR AND DSND TO 5000 FT. ... FURTHER CHOKING... SHOWED TERRAIN AT 
1 4000 FT TO 1 1000 FT DIRECTLY ALONG OUR PATH. A SIMILAR ATC CLRNC HAPPENS VERY OFTEN FLYING INTO LIMA, 
PERU. MANY, MANY, MANY PLTS ARE NOT AWARE OF JUST HOW CRUCIAL IT IS NOT TO ACCEPT THESE DEADLY 
CLRNCS. PLEASE GET THE WORD OUT AGAIN. 


140711 B ARRANQUELLA CTL CLRED US DIRECT TULUA VOR. AS WE WERE PASSING ABEAM CARTAGENA VOR, AN LGT 
Y CROSSED OUR NOSE HDG IN A NE DIRECTION. IT WAS EXTREMELY CLOS E AND WE ARE SURE HE WAS AT OUR ALT 
BECAUSE WE HIT HIS WAKE TURBULENCE AS WE PASSED BEHIND HIM. ... B ARRANQUELLA CTL STATED THAT HE WAS 
CTLING NO OTHER ACFT IN OUR AREA AT FL330. IN THE FUTURE, I WILL NOT ACCEPT AN OFF AIRWAYS CLRNC WHEN 
NOT POSITIVE OF BEING IN RADAR CONTACT. CILR STATED THAT SOMETIMES ACFT TRANSVERSE HIS AIRSPACE THAT 
HE WAS NOT CTLING. 

349669 I BELIEVE THIS WAS A CLASSIC SIT OF A FOREIGN ATC LANGUAGE BARRIER. ...I HAD NEVER BEEN TO 
MANAGUA BEFORE... SHE SAID RPT 5 DME RWY 9.L. WE BOTH ASSUMED WRONGLY THAT SHE HAD MEANT RPT 5 
DME OUT ON FINAL ON THE APCH TO RWY 9. ... AS WE WERE TURNING FINAL AT APPROX 9 DME FROM THE RWY THE 
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CTLR ASKED OUR POS AND ALT. WE RESPONDED WITH, ON FINAL, 9 DME AT 2700 FT AS PUBLISHED.' SHE THEN SAID 
THAT WE WERE TO HAVE C ALLED 5 DME FROM THE VOR AND CROSSED THE VOR AT 5000 FT. 

334006 TCASII SCREEN SHOWED TFC 12 O'CLOCK, 800 FT BELOW US AND CUBING! ... I TURNED THE ACFT R AT THE 
LAST MOMENT BECAUSE WE FINALLY SAW NAV LIGHTS OF THE OTHER ACFT AT OUR 11:30 POS. ...I HAVE BEEN 
FLYING THIS AIRSPACE FOR OVER 5 YRS NOW AND WITH THE HORRENDOUS ATC (AND/OR UNSCRUPULOUS LATIN 
AMERICAN OPERATORS FLYING ANY ALTS THEY WANT) THERE WILL BE A MIDAIR COLLISION BEFORE TOO LONG 
WE NEED, AND THE TRAVELING PUBLIC SHOULD DEMAND, SATCOM AND BETTER ATC THAN THIS ARCHAIC LATIN 
AMERICAN SYS, NOTE ALL THE LATA AIRWAYS ON MAPS WHERE ATC IS OF DUBIOUS QUALITY ... 

4.2. Problems with operations in other foreign locations (2) 

Incident 244767 is reminiscent of the August 1997 crash of Korean Air Flight 801 on Guam (McKenna, 1998b). 

244767 SCHEDULED PART 121 FLT FROM SPN (SAIPAN) TO MANILA, PI (MNL). WX AT THE TIME WAS RPTED 1 KM TO 1 
IP KM IN HVY RAINS... COMS DIFFICULT AT BEST DUE TO POOR EQUIP AND CTLR'S HVY ACCENT. ... ATIS INFO NEVER 
SPECIFIED APCH IN USE OR IF ANY PART OF ANY APCH SYS WAS NOT IN SVC. APCH CTLR STATED APCH TO RWY 24 EM 
USE ' WE (ALL 3 PLTS) BRIEFED THE ILS 24 APCH ... I STATED SEVERAL TIMES THE ILS FREQ IS NOT IDENTCMG. THIS, 
SAD TO SAY, IS NOT UNUSUAL IN MANILA DUE TO VERY POOR GND EQUIP. CTLR VECTORED US ON THE INTERCEPT 

HDG AND THEN CLRED US FOR THE RWY 24 APCH.' THE CAPT (PF) CAPTURED THE LOC AND GS AND BEGAN TO LET 

DOWN. AT MINIMUMS, ARPT WAS NOT IN SIGHT SO A MISSED APCH WAS EXECUTED. TO OUR DISMAY, THE OMEGA 
READ 7 MI FROM THE ARPT. WE WERE 7 MI NE OF THE ARPT AT 400 FT! 

305840 APCH CTL VECTORED US W THROUGH FINAL AND THEN BACK. THE APCH WAS NOT IN THE FMS DATABASE 
AND BOTH PLTS WERE USING RAW DATA. AS WE SAW THE DEV BAR BEGIN TO CTR, THE CAPT (WHO WAS HAND 
FLYING) CALLED ARPT IN SIGHT.' IT WAS APPARENT THAT WE WERE TOO HIGH AND A FEW SECONDS (5- 1 0) LATER THE 
FO AND I REALIZED THIS WAS NOT THE NAGOYA ARPT. THE VISUAL APPEARANCE WAS SIMILAR, BUT RWY HDG WAS 
30 DEGS OFF AND NOW WE WERE E OF THE FINAL APCH COURSE, AND STILL APPROX 9 MI NNE OF NGO. LANGUAGE 
AND PHRASEOLOGY WERE DEFINITELY A FACTOR. IT WAS NOT QUITE CLR, AFTER SEVERAL QUERIES, IF WE WERE 
CLRED FOR THE APCH OR WERE STILL ON VECTORS. A HEADS UP CALL THAT ONE MIGHT EXPECT IN CONTINENTAL 
UNITED STATES SUCH AS ARPT AT 2 O'CLOCK, 9 ML'IS NOT USED IN JAPAN OR OTHER ASIAN COUNTRIES THAT I'M 
AWARE OF. THE CAPT WAS NEW TO INTL FLYING AND TO THE ACFT (4 MONTHS APPROX) AND SAW WHAT HE 
EXPECTED TO SEE AND MADE a QUICK DECISION BASED ON LIMITED INFO. 
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Appendix 7. 

Excerpts of each of the 16 incidents 
that were rated relevant by QUORUM but not by the analysts 




Appendix 7. Excerpts of each of the sixteen incidents that were rated relevant by QUORUM but not by the analysts. The 
16 incidents fall into 9 categories. The first two incidents involve GPWS alarms, and are clearly relevant. The incident 
involving an escape maneuver is also relevant. The incident describing problems and confusion with terminal area charts 
seems somewhat relevant, as does the incident involving speed brakes. The six incidents involving FMS and approach, 
or approach/descent, are only vaguely relevant. Three incidents involving ACCIDENT are not relevant, nor are the two 
incidents involving CIVIL or FAA/SAFETY. So, of the 16 incidents rated as irrelevant by the analysts but rated as relevant 
by QUORUM, 5 are clearly irrelevant, 6 are vaguely relevant, and 5 are relevant. 

1 . GPWS alarms 

Since this incident involved an event that also occurred in the Cali accident, a GPWS alarm, this incident is relevant. Further, it 
illustrates crew reluctance to respond to the GPWS when it is perceived to be giving a false alarm. This issue is clearly relevant 
to CFIT accidents. Still, one analyst strongly disagreed that incident 228422 has any relevance or similarity to the Cali 
accident. 


228422 THE GPWS GAVE REPEATED TERRAIN, TERRAIN, WHOOP, WHOOP, PULL UP WARNINGS. SINCE WE WERE IN 
DAY VMC FLT CONDITIONS AND COULD CLRLY SEE THAT WE WERE NOT IN DANGER OF A GND COLLISION, WE DID NOT 
PULL UP. AFTER REPEATED WARNINGS, WE SELECTED THE GPWS OVERRIDE TO SILENCE THE AURAL WARNING. ... 
THIS SAME TYPE OF INCIDENT HAS OCCURRED WHILE FLYING THE LOC/DME BACK COURSE RWY 8 APCH AT 
MARTINSBURG, WV, AND WHILE BEING VECTORED AT ROANOKE VA. ... TO BE REQUIRED TO EXECUTE AN ABRUPT 
PULL UP WHILE FLYING A PUBLISHED PROC DOES NOT ENHANCE SAFETY. ON THE OTHER HAND, THE RECORD WILL 
SHOW THAT THERE HAVE BEEN A NUMBER OF ACCIDENTS THAT OCCU RRED BECAUSE FLCS ELECTED TO IGNORE THE 
GPWS AND CONSEQUENTLY COLLIDED WITH THE GND. THE DESIGN CRITERIA FOR THE GPWS AND THE TERMINAL 
INSTS PROCS SHOULD BE CORRELATED TO PREVENT THESE FALSE WARNINGS. 


Incident 280922 involved an event that also occurred in the Cali accident, a GPWS alarm, so this incident is relevant. 

280922 BELOW 1000 FT WE GOT A GPWS TOO LOW TERRAIN’ AND TOO LOW GEAR' WARNING AT THE SAME TIME THE 
TCASII WAS TELLING US TO DSND. ... DEP CTL THEN CALLED TO INQUIRE ABOUT OUR CLB OR LACK OF CLB AT ABOUT 
THE SAME TIME THE GPWS BURPED TOO LOW TERRAIN' OR TERRAIN, TER RAIN. ' THE CONFUSION AND TENSION IN 
THE COCKPIT WAS QUITE HIGH ... THE CAPT AND I AGREED THAT UNDER DIFFERENT CIRCUMSTANCES, S AY A LOW 
VISIBILITY TKOF OR AN ENG FAILURE AFTER TKOF, THAT THE CONFLICTING COMMANDS ('DSND, DSND7TERRAIN, 
TERRAIN') COULD EASILY CAUSE AN ACCIDENT. 

2 . Escape maneuver 

Since incident 313511 involved an event that also occurred in the Cali accident, an escape maneuver, albeit in response to a 
windshear alarm, this incident is relevant. In fact, the Colombian report on the Cali accident stressed the relationship between 
windshear training and GPWS training: 

"Simulator training is the best method for pilots to extract maximum performance from large 
airplanes during a CFIT escape maneuver. Therefore, Aeronautica Civil urges the FAA to require a CFIT 
training program that includes realistic simulator exercises comparable to the successful windshear 
and rejected takeoff training programs." 


The NTSB agreed: 

"Develop a controlled flight into terrain training program that includes realistic simulator exercises 
comparable to the successful windshear and rejected takeoff training programs and make training in 
such a program mandatory for all pilots operating under 14 CFR Part 121. (Class II, Priority Action) 
(A-96-95)" 


Thus, incident 313511 is particularly relevant to the Cali accident. 

313511 WITHOUT ANY NORMAL WARNINGS (AIRSPD LOSS OR GAIN, SINK RATE CHANGE, ETC) WE WERE SUDDENLY 
GIVEN AN ON-BOARD WINDSHEAR’ ALERT (AURAL AND RED LIGHT). I IMMEDIATELY EXECUTED AN ESCAPE 
MANEUVER (MAX FIREWALL PWR, 20 DEGS NOSE UP) AS ACFT BEGAN DSNDING. ... I AM ONLY GLAD THAT I JUST 
COMPLETED ANNUAL RECURRENT TRAINING IN WHICH WINDSHEAR TRAINING WAS AN EMPHASIS ITEM. I DID NOT 
HESITATE TO INITIATE THE ESCAPE MANEUVER, BUT I’M NOT CERTAIN THAT I WOULD HAVE DONE SO IMMEDIATELY IF 
I HADNT JUST HAD THE NEED FOR IMMEDIATE AND DECISIVE RESPONSE EMPHASIZED IN THAT TRAINING!! 
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3 . Charts and confusion in the terminal area 


Incident 112422 raises questions about the clarity and adequacy of some terminal area charts. The themes of confusion in the 
terminal area and confusion regarding charts also appear in the Cali accident. Thus, this incident is relevant. Perhaps the fact 
that this incident involved a general aviation pilot convinced the analysts that it was not relevant. 


1 12422 I WAS INBND TO SAN DIEGOS MONTGOMERY FIELD FROM SANTA BARBARA VIA VAN NUYS, POMONA, AND 
OCEANSIDE VORS. ... I ASKED THE CTLR TO VERIFY THAT I HAD CLRNC INTO THE TCA. THE RESPONSE WAS "YOUVE 
BEEN IN THE TCA FOR SEVERAL MINUTES ' .. SINCE THE SAN DIEGO TCA CONSTITUTES THE MOST CONFUSING MAZE 
OF AIRSPACE I HAVE TO DATE ENCOUNTERED, I HAD CONDUCTED VERY CAREFUL FLT PLANNING TO AVOID LAX TCA 
AND TO PLAN FLT INTO MONTGOMERY ON ROUTES AND ALTS AVOIDING THE SAN TCA. IN THE COURSE OF THAT FLT 
PLANNING I OBSERVED SEVERAL CRITICAL OMISSIONS ON THE SAN VFR TERMINAL AREA CHART. THE QUEST FOR 
"ALL AVAILABLE INFO" REQUIRED BY FAR 91.5 SHOULD BE AIDED, NOT HINDERED, BY THE CHART-MAKERS, AS 
FOLLOWS: ... 

4 . Speed brakes 


Incident 340978 centers on a problem with speed brakes that began during descent. Speed brakes also played a central role in the 
Cali accident. On that basis, this incident is relevant. It is reasonable to suggest that an accident investigator might want to 
explore speed brake incidents to see if there are any operational difficulties that might be related to the Cali accident. As 
indicated by incidents 334866 and 280233 in appendix 6, section 2.3., "Forgot speed brakes," the use of speed brakes is a 
topic worthy of investigation. Thus, reports of problems with speed brake have some relevance. 

340978 ON DSCNT INTO SFO I TRIED TO ARM THE SPD BRAKES BUT COULD NOT GET A GREEN SPD BRAKE ARMED 
LIGHT, JUST THE SPD BRAKE DO NOT ARM LIGHT. SO I STOWED THE SPD BRAKE LEVER IN THE DOWN DETENT POS (NO 
LIGHTS). WE USED FLAPS 40 DEGS FOR THE FLAP SETTING. I PLANNED A LONG ROLLOUT (ANTI-SKID INOP, MANUALLY 
SPD BRAKE, REDUCED BRAKING). ... WHEN I TRIED TO USE THE REVERSER THEY WOULD NOT DEPLOY, IT TOOK A 
COUPLE OF TREES . THEN WHEN THEY DID DEPLOY I NOTED THAT THE SPD BRAKE HANDLE MOVED TO THE UP POS B Y 
ITSELF AND AT THAT POINT IN TIME WHAT SEEMED TO ME AS BRAKING Ol * STRONG DECELERATION , ALL THIS WITH 
VIOLENT SHAKING. BRAKES WERE NOT BEING USED BY ME OR THE FO. 

5 . Use of automation on descent/approach 


Incident 308422, like the Cali accident, involves letting automation fly the aircraft in a demanding environment during 
descent/approach. While the turn in this incident was intended, both the incident and the accident involved safety concerns 
associated with turns initiated by automation. This incident is vaguely relevant to the Cali accident. 


308422 I WAS LETTING THE FMS AND AUTOPLT FLY THE ARR. ... THE FMS STARTED THE TURN FROM DARTS TO FINAL 
APCH APPROX 2 MI PRIOR TO DARTS. THE FMS IS DESIGNED TO DO THIS SC* THAT THE ACFT WILL NOT OVERSHOOT THE 
NEW COURSE. ... SOCAL APCH [said] THAT WE HAD INTRUDED INTO ANOTHER CTLR S AIRSPACE BY STARTING THE 
TURN EARLY AND AD VISED THAT WE NOT LET THE FMS FLY THE AIRPLANE IN LOS ANGELES BASIN. 


6 . On approach 


These incidents are, at best, only vaguely relevant to the Cali accident. They involve some sort of problem while on approach, 
but have little else in common with the accident. 


Incidents 370656 and 332870 describe parallel approaches to SFO. QUORUM picked up that there was a problem on approach, 
but misinterpreted the contextual association of "FMS" (flight management system) and "APCH" (approach) in the narrative. 


370656 WHILE SHOOTING THE FMS BRIDGE VISUAL RWY 28R APCH TO SFO, APCH CTL C ALLED OUR TFC AT 10 
O'CLOCK HIGH AND THAT IT WOULD BE GOING TO RWY 28L. ... WE ACKNOWLEDGED THE TFC AND CONTINUED ON THE 
FMS APCH PASSING THE GAROW FIX (15 DME AT OR ABOVE 4000 FT) AS PRESCRIBED. ...WE THOUGHT IT ODD THAT HE 
WAS CRUISING RIGHT BY US, BUT FIGURED WE WOULD HAVE THE REQUIRED 1/8 MI STAGGER PRIOR TO TOUCHDOWN. 

332870 SFO APCH CLRED US THE FMS RWY 28R APCH ... THEY THEN ADVISED US A B737 WOULD BE MAKING AN 
APCH TO RWY 28L. WE KEPT THE B737 AHEAD OF US BUT BARELY. THE REASON FOR THIS RPT IS THE APCH UPSET 
SEVERAL PAX ON THE L SIDE OF THE ACFT. THEY THOUGHT WE WERE WAY TOO CLOSE. I KNOW THIS INCREASES ARR 
RATES, BUT IT CAN BE UPSETTING TO PAX. 
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Incident 281636 happened during approach, but has little else in common with the Cali accident other than the crew having a 
problematic interaction with the controller. QUORUM over-interpreted the presence of the term "FLC" (flight crew) in the 
context of "APCH" (approach). 


281636 ON THE QUIET BRIDGE VISUAL TO SFO RWY 28R APCH CTL CALLED OUT AN ACFT AHEAD. ... CTLR THEN 
CLRED OUR ACFT TO MAINTAIN 2500 FT AND 250 KTS. OUR ACFT WAS NOW APPROX 7 DME FROM RWY 28R. WE 
QUERIED THE CTLR ABOUT THE AIRSPD HE WANTED US TO MAINTAIN AND HE REPLIED HIS INTENTIONS WERE TO HAVE 
US OVERTAKE THE ACFT THAT WAS AHEAD, BELOW AND TO OUR L... AT APPROX 6.5 DME WE TOLD THE CTLR THAT 
UNLESS WE COULD REDUCE OUR AIRSPD WE COULD NOT LAND. CTLR REPLIED BY HAVING US DISCONTINUE THE 
APCH. WE INFORMED THE CILR THAT WE NOW HAD THE COMMUTER ACFT IN SIGHT AND THAT IT WAS ABOVE US AND 
BEHIND US. CTLR RESPONDED ROGER, MAINTAIN VISUAL WITH THE COMMUTER ACFT AND SWITCH TO TWR FREQ. WE 
SWITCHED OVER TO SFO TWR CTL AND INFORMED THE CTLR THAT WE COULD NOT MAINTAIN VISUAL WITH AN ACFT 
THAT WAS ABOVE AND BEHIND US. ... APCH CTLS DIRECTION TO HAVE A FLC MAINTAIN VISUAL WITH AN ACFT THAT 
IS ABOVE AND BEHIND THEM WHILE ON FINAL APCH IS NOT ONLY IMPRACTICAL, BUT IMPOSSIBLE AND DANGEROUS 
IF ATTEMPTED. 


Incident 325365 happened during approach, but has little else in common with the Cali accident other than the crew 
misunderstanding the controller. QUORUM over-interpreted the presence of the term "FLC" (flight crew) in the context of 
"APCH" (approach). 

325365 WE HAD ... BEEN GIVEN A NORTHERLY VECTOR FOR A VISUAL APCH TO LOS ANGELES INTL ARPTS S 
COMPLEX. ... A JETLINER, ALSO FOR THE S COMPLEX, WAS APCHING LAX FROM THE E. ... THE CTLR ISSUED A VISUAL 
APCH CLRNC INTENDING FOR US TO MANEUVER OVER AND BEHIND HIS PATH FOR RWY 25R WHILE HE PROCEEDED TO 
RWY 25L.... WE MAY HAVE INADVERTENTLY ANTICIPATED THE CTLR'S INSTRUCTIONS AND PREPROGRAMMED 
OURSELVES TO HEAR AND BELIEVE THAT WE HAD BEEN CLRED FOR THE RWY 25L VISUAL APCH. ... HE BELIEVES THEY 
WERE NEVER CLOSER THAN 1/4 MI FROM THE B767. HE TALKED TO SOCAL APCH CTLR WHO INDICATED THE ACR FLC 
WAS UPSET BUT SOCAL INDICATED THEY WERE ON A VISUAL APCH AND SOCAL WAS NOT RESPONSIBLE FOR 
SEPARATION. 

In the narrative describing incident 260432, the reporter provides a lengthy description of the details of a normal approach as a 
set-up for the sudden, abnormal, and violent roll excursions. QUORUM misinterpreted the detailed set-up as central to the 
incident. Mention of the flight data recorder also appeared to QUORUM to signify relevance. 


260432 WE INTERCEPTED FINAL ABOUT 5 MI OUTSIDE THE FINAL APCH FIX AND PROCEEDED TO FLY A NORMAL APCH 
USING THE FMS/TLS AND STANDARD ACR PROCS. AN APCH CHK WAS RE-ACCOMPLISHED TO IDENT THE RWY 36R LOC 
AND CONFIRM THE NEW MINIM UMS FOR THAT APCH. ... AT ABOUT 500 FT AGL, THE COPLT DISCONNECTED THE 
AUTOPLT, AND BEGAN TO HAND-FLY THE ACFT, WITH THE AUTO THROT TLES STILL CONNECTED. ... LATER ANALYSIS 
OF THE FLT DATA RECORDER DID NOT SHOW ANY DISCERNIBLE DIFFERENCE BTWN THE AUTOPLT AND COPLT FLYING. 
... AS THE RWY THRESHOLD LIGHTS WERE PASSING UNDER THE NOSE, THE ACFT ENTERED INTO A SERIES OF ABRUPT 
AND VIOLENT ROLL EXCURSIONS WHICH I ESTIMATE TO BE IN THE RANGE OF 15-20 DEGS OF BANK THERE WERE 3 OR 
4 OF THESE ROLL REVERSALS, WHICH ENDED AS ABRUPTLY AS THEY BEGAN. AT THIS POINT, I WOULD ESTIMATE THE 
ACFT ALT AT ABOUT 15 FT... 


7 . "ACCIDENT" in the context of "PLTS" or "TRAINING" 

The words "ACCIDENT" and "PLTS" are rare in the ASRS database, but are much more common in the Cali documents. Once 
QUORUM had already identified the more obviously relevant reports, it over-interpreted the relevance of those, like incidents 
342160 and 360500, having words such as "ACCIDENT" in the context of "PLTS" (pilots). 

342160 WE DO AERIAL FIRE-FIGHTING - OUR MISSION IS LOW LEVEL AND VERY INTENSE FLYING. I AM CONCERNED 
NOT ONLY FOR OUR AGENCY PLTS, BUT ALSO FOR THE CONTRACT AIR TANKER PLTS. IF YOU CHK OUR ACCIDENT 
RECORD, YOU'LL SEE WHY I'M CONCERNED. WE ARE REQUIRED TO WORK EITHER 12 DAYS ON AND 2 DAYS OFF OR 6 
DAYS ON AND 1 DAY OFF - OUR DUTY DAYS AVERAGE 10-12 HRS. 

360500 LCLPLTS PERSISTENTLY TAXI ON RWY 04/22 ATTHE INTXN OFTXWY A WHEN INSTRUCTED TO TAXI TO RWY 
29. TRANSIENT PLTS WILL REMAIN CLR OR HOLD SHORT OF RWY 04/22 UNLESS AUTH BY ATC. THIS IS AN ACCIDENT 
WAITING TO HAPPEN AT THIS FACILITY BECAUSE THE ONLY PLTS THAT ACCESS RWY 04/22 WHEN INSTRUCTED TO 
TAXI TO RWY 29 ARE LCL PLTS ... 
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Incident 325432 looked relevant to QUORUM primarily because of the proximity of "ACCIDENT" and "TRAINING." This 
pairing is rare in the ASRS database and more common in the Cali documents. 


325432 I DEPARTED FROM FT LAUDERDALE EXECUTIVE ARPT WITH MR X, AN FAA SAFETY INSPECTOR/PLT ON A 
PART 141 TRAINING FLT. ... THIS TRAINING PERIOD WAS TO INCLUDE TOUCHDOWN AUTOROTATIONS. ... THE FAA HAS 
MADE THIS MANEUVER A REQUIRED ITEM IN THEIR CONTRACT DEMANDS FOR OUT OF AGENCY RECURRENCY 
TRAINING. ACCORDING TO PRELIMINARY ACCIDENT RPTS AS COLLECTED AND CIRCULATED TO THE HELI INDUSTRY, 
THE MOST COMMON TRAINING ACCIDENT INVOLVES THE PRACTICING OF TOUCHDOWN AUTOROTATIONS. 

8 . A "CIVIL” incident 

Incident 148439 looked relevant to QUORUM primarily because of the presence of the word "CIVIL" in numerous QUORUM 
relations. The word "CIVIL" is rare in the ASRS database and common in the Cali document (due to references to the Colombian 
agency investigating the accident, Aeronautica Civil, and the International Civil Aviation Organization). 


148439 SEVERAL CALLS WERE MADE TO THE TANKER REQUESTING A RFT OF HIS INTENTIONS, BUT IT SEEMED THAT 
THE TANKER WAS NOT ON FREQ. LATER I WAS ABLE TO CONTACT THE TANKER PLTBY PHONE AND LEARNED FROM 
HIM, TO MY GREAT SURPRISE, THAT HIS ACFT IN NOT EVEN EQUIPPED WITH THE VHF RADIOS NECESSARY TO 
COMMUNICATE WITH CIVIL AIR TFC! ... THE MIL WOULD LOOK ESPECIALLY BAD IF INVESTIGATORS DISCOVERED 
THAT THE CIVIL ACFT HAD BEEN IN COMPLIANCE WITH FAA RECOMMENI >ED RADIO PROCS AND THE MIL ACFT HAD 
NOT, BECAUSE OF HIS LACK OF BASIC RADIO EQUIP. 

9 . "FAA" and "SAFETY" 

In the ASRS database, the words "FAA" and "SAFETY" usually do not describe operational details of incidents, but are 
occasionally found in general opinions that some reporters add to their narratives. In the context of accident investigations, 
however, these words are much more common, as in the Cali documents. QUORUM misinterpreted the prominence of relations 
containing "FAA" and/or "SAFETY" as an indication that incident 355188 is relevant to the Cali accident. 

355188 WE WERE ON FAA PROVING RUNS ENRTE FROM IAH TO MCO. ... UNFORTUNATELY, OUR FAA AVIATION 
SAFETY INSPECTOR HAD SPECIFICALLY REQUESTED THAT WE DO NOT ADVISE ATC OF THEIR SIMULATED EMER/ BUT 
THAT WE WERE TO IMPROVISE PROCS TO EXPEDITE A LNDG. IN AN ACR AL EMER SIT SUCH AS THIS, ATC IS OUR 
PRIMARY AND MOST IMPORTANT RESOURCE, A RESOURCE DENIED TO US BY OUR AVIATION SAFETY INSPECTOR. ... 
DURING THIS DSCNT, I BRIEFED THE CREW FOR A LNDG AT DOWNTOWN ARPT, WITH THE FAA AVIATION SAFETY 
INSPECTOR OBSERVING THE BRIEFING. 
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